QlikView

Processing with QLIK

Issues With CPU in QLIK

--> Not able to utilize full performance of CPU

1) Description:-

The number of CPU affinity for the QVS does not match the CPU cores in the Server.

   Cause:-

This issue comes from a new BIOS setting introduced in the Generation 9 of HP servers.
A new setting has been added in BIOS that needs to be modified to allow the QlikView services properly detect all the CPU Cores in the machine.

  Resolution:-

This issue can be resolved by changing the BIOS setting "NUMA Group Size" to "Flat".
The default setting for "NUMA Group Size" disables QlikView from properly detecting all
CPU cores in the hardware, and thereby
From HPs manual on the subject: "Use this option to configure how the system ROM reports
  the number of logical processors in a NUMA (Non-Uniform Memory Access) node. The
  resulting information helps the operating system group processors for application use. To set             

NUMA Group Size Optimization:

   From the System Utilities screen,
   select System Configuration →
   BIOS/Platform Configuration (RBSU) →
   Performance Options →
   Advanced Performance Tuning Options →         
   NUMA
  Group Size Optimization and press Enter.
  Select a setting and press Enter:

Clustered (default)—Optimizes groups along NUMA boundaries, providing better performance.
Flat—Enables applications that are not optimized to take advantage of processors spanning                  multiple groups to utilize more logical processors."


======================================================================

2) Description

NUMA, Non-Uniform Memory Access, is a computer memory design used in Multiprocessing, where the memory access time depends on the memory location relative to a processor. Under NUMA, a processor can access its own local memory faster than non-local memory, that is, memory local to another processor or memory shared between processors.

QlikView generally performs better with NUMA disabled. The NUMA setting is controlled by the computer BIOS, and the recommendation is to disable it for best possible performance.

Some hardware does not allow disabling NUMA. For these scenarios QlikView 11.20 SR2 and later can automatically detect the presence of NUMA and if present adjust the QVS.exe to adapt to a NUMA environment. While not as efficient as disabling NUMA in BIOS, it restores some of the performance lost when NUMA is enabled.

The automatic detection only works in 2 or 4-socket systems. For 8 socket machines the NUMA detection fails, leaving the QVS.exe to believe itself to exist in a non-NUMA environment.

The software NUMA adaption is controlled with the EnableNumaRoundRobin variable in the QlikView Server settings.ini file.

From the tests which Performance & Scalability team is doing on new releases they have not seen any NUMA specific changes in QV 12.10 and 12.20.

The recommendations can be found on community: https://community.qlik.com/docs/DOC-2362 they are being updated if Performance & Scalability team sees that changes in server configurations impacts the performance. So it worth checking them from time to time.

Please keep in mind that Performance & Scalability team tries to test many hardware configurations, but it is still a very small subset of possible server configurations seen in customer deployments. Due to this, in case of doubt they suggest to test the settings impact. For example one customer might on average see that for their deployment disabling NUMA in BIOS is beneficial, while other might not notice any difference.

Resolution:-

EnableNumaRoundRobin

0 (default): Automated state where QlikView Server at startup detects whether NUMA is enabled or disabled at certain hardware and adapts to it. If NUMA is enabled QlikView Server will ignore NUMA nodes when allocating memory.
1: QlikView Server  is acting as if NUMA is enabled from BIOS (without any attempts to detect it)
2: QlikView Server will not ignore NUMA nodes if present (i.e. NUMA enabled)

The Settings.ini file can be found in the following location:

Windows Server 2003
C:\Document and Settings\All Users\Application Data\QlikTech\QlikViewServer
Windows Server 2008 and later
C:\ProgramData\QlikTech\QlikViewServer for Windows 2008 and later





========================================================================

3)  QlikView and NUMA 

NUMA technology typically provides performance gains over other multiprocessing
architectures especially if a software application is designed to take advantage of NUMA. However, extensive
tests at our Scalability Center prove that NUMA-enabled servers have a negative impact on QlikView Server
performance. This impact is less noticeable on systems with fewer CPU sockets as the need to access remote
memory is reduced. Therefore, for NUMA architectures, it is beneficial to disable NUMA in the BIOS settings of
the hardware running the QlikView Server in order to achieve optimum performance. Some vendors have removed
the ability to turn off NUMA in the BIOS. If that is the case, a soft-switch has been implemented into the QlikView
configuration which circumvents NUMA memory allocation. Since QlikView 11.2 SR2, QlikView will automatically
detect if NUMA has been disabled and adapts appropriately for optimal performance. (For QlikView releases prior
to QV11.2 SR1, see this document for more information on this setting: http://community.qlikview.com/docs/DOC-23
62) If a QlikView application (.QVW) is opened and loaded into memory on a NUMA-enabled system with the
soft-switch logic disabled, the operating system will use the local memory for one CPU at a time. When necessary,
it will use remote memory within another NUMA node (See Figure 1). (CPU access time to its local memory is

much faster than access to remote memory). QlikView scales well over multiple cores resulting in those cores
needing access to the same memory if “everything” is stored in one place. Since QlikView is not optimized for
NUMA, please take note of the soft-switch setting for QlikView versions prior to 11.2 SR2 or disable NUMA in
the BIOS if applicable.

========================================================================

Servers running QIX engine benefit from the following settings on average:

Setting/Configuration	Value
Hyper-threading	Enabled 4 socket /Disabled 2 socket Note: For 4-socket solutions, having hyper-threading enabled is beneficial on average. There are, however, cases when disabling it leads to improved performance.
Power Management (Power profile)	Max performance (BIOS and Windows setting)
Node Interleaving	Enabled (meaning that NUMA is disabled) Note: Disabling NUMA has no significant impact on the performance as the product adapts its behavior if NUMA is enabled.
Intel Turbo Boost	Enabled
Memory configuration	Configured for best performance Note: The DIMM slots for every CPU should be populated in accordance to the hardware manufacturer’s specification for best performance.
Hardware pre-fetch	Enabled

=======================================================================
Test done by users:-

Scheduled 40 concurrent QVW reloads in the QDS (the QVW's generated data, wrote QVDs, read QVDs, and then logged stats).

The test hardware was a HP DL580 G7 with 4 X Intel Xeon Processor E7-4870 (4 CPU X 10 cores) and 256GB RAM.  By disabling hyperthreading, the average document refresh time went from 02:53 to 01:50.

=============================================================

Conclusions:-

-->Preferably we need to have utmost 2 sockets of CPU cores at the Max for QLIK.
-->Disabling NUMA is Advised if performance is not good.
-->If you don't have option to disable then do soft switch enable.
( if in case NUMA cannot be disabled, "EnableNumaRoundRobin=0" has to be written in "setting.ini" under "QlikViewServer" folder.
EnableNumaRoundRobin=0 is defaulted and means disabled functionality.

EnableNumaRoundRobin=1 means enable functionality which should be used for servers where NUMA cannot be disabled in the hardware)

-->Hyper-threading should be turned off to achieve maximum CPU Utilization.