This is the mail archive of the
systemtap@sourceware.org
mailing list for the systemtap project.
Re: Poor scaling
- From: David Smith <dsmith at redhat dot com>
- To: Avi Kivity <avi at scylladb dot com>, systemtap at sourceware dot org
- Date: Mon, 8 Aug 2016 10:00:14 -0500
- Subject: Re: Poor scaling
- Authentication-results: sourceware.org; auth=none
- References: <77356e45-d67b-6836-510c-a10aa23347b4@scylladb.com>
On 08/07/2016 07:47 AM, Avi Kivity wrote:
> Running a process with 12 active threads, each bound to its own core,
> and touching a systemtap array fairly frequently, I see:
>
>
> 72.91% scylla [kernel.kallsyms] [k] _raw_spin_lock
> 3.49% scylla [kernel.kallsyms] [k] start_callback.isra.62
> 1.77% scylla scylla [.]
> _ZSt13__adjust_heapIN9__gnu_cxx17__normal_iteratorIPPN8logalloc11region_implESt6vectorIS4_SaIS4_EEEElS4_NS0_5__ops15_Iter_comp_iterIZNS2_7tracker4impl15compact_on_idleERKSt8functionIFbvEEEUlS4_S4_E_EEEvT_T0_SM_T1_T2_.isra.437.constprop.582
>
> 1.54% scylla [kernel.kallsyms] [k] _raw_spin_unlock
> 1.39% scylla scylla [.]
> _ZN7reactor10smp_pollfn4pollEv
> 1.29% scylla libc-2.19.so [.] epoll_pwait
> 1.21% swapper [kernel.kallsyms] [k] poll_idle
> 1.06% scylla [kernel.kallsyms] [k] _raw_read_lock_irqsave
> 1.01% scylla scylla [.]
> _ZSt11__push_heapIN9__gnu_cxx17__normal_iteratorIPPN8logalloc11region_implESt6vectorIS4_SaIS4_EEEElS4_NS0_5__ops14_Iter_comp_valIZNS2_7tracker4impl15compact_on_idleERKSt8functionIFbvEEEUlS4_S4_E_EEEvT_T0_SM_T1_T2_.isra.436.constprop.583
>
> 0.80% scylla [kernel.kallsyms] [k]
> _raw_read_unlock_irqrestore
> 0.78% scylla scylla [.]
> _ZN17smp_message_queue20flush_response_batchEv
> 0.75% scylla [kernel.kallsyms] [k]
> utrace_report_syscall_entry
> 0.63% scylla [kernel.kallsyms] [k] task_utrace_struct
> 0.50% scylla [kernel.kallsyms] [k] native_read_tsc
> 0.37% scylla [kernel.kallsyms] [k]
> utrace_report_syscall_exit
> 0.32% scylla [kernel.kallsyms] [k] delay_tsc
> 0.28% scylla scylla [.]
> _ZN17smp_message_queue13process_queueILm2EZNS_16process_incomingEvEUlPNS_9work_itemEE_EEmRNS_8lf_queueET0_.isra.2371
>
> 0.26% scylla scylla [.]
> _ZN17smp_message_queue13process_queueILm4EZNS_19process_completionsEvEUlPNS_9work_itemEE_EEmRNS_8lf_queueET0_.isra.2278
>
> 0.20% scylla [kernel.kallsyms] [k] _raw_spin_lock_irq
> 0.18% scylla scylla [.]
> _ZN6memory9cpu_pages4freeEPv
> 0.18% scylla liblz4.so.1.0.0 [.] LZ4_compress
> 0.17% scylla scylla [.]
> _ZN7reactor9run_tasksER15circular_bufferISt10unique_ptrI4taskSt14default_deleteIS2_EESaIS5_EE
>
>
>
> Now this is not something I can run to monitor a production load.
>
>
> Is there any way I can improve scaling?
It depends, we'd have to see your script to make suggestions.
--
David Smith
dsmith@redhat.com
Red Hat
http://www.redhat.com
256.217.0141 (direct)
256.837.0057 (fax)