Throughput drop with new DPDK 22.11.2 with QAT gen3 c4xxx

[sai19]

We upgraded DPDK from 20.11.2 to DPDK 22.11.2 and we use symmetric crypto with QAT gen3 c4xxx. Seeing 25% drop of throughput with this qat gen 3 with dpdk 22.11.2. We are using gat gen2 in different platform - we are not seeing any issue there with dpdk 22.11.2, where we are getting same throughput as dpdk 20.11.2.

The only suspect I can see is this commit https://git.dpdk.org/dpdk-stable/commit/?h=v22.11.2&id=254558c832d7b35faf2aedb3b324d04d844bedec which introduces using qat_sym_convert_op_to_vec_aead which is called from qat_sym_build_op_aead_gen3 and which in turn calls rte_crypto_mbuf_to_vec. Could some one provide explanation what exactly these functions   qat_sym_convert_op_to_vec_aead and  rte_crypto_mbuf_to_vec actually does. Are these functions adding more overhead and consuming more time and hence causing throughput drop? 

With dpdk 22.11.2, does qat gen3 requires some tuning for maximum performance, do we need to enable some flags. Could anyone help with these couple of queries.

Help is much appreciated, Thanks for the help in advance.

[Ronny_G_Intel]

Hi sai19,

While I am researching and looking for answers to your questions, can you please provide me with information about the hardware that you are using the QAT driver package version?

Regards,

Ronny G

[sai19]

Thanks a lot for responding to this query!

We are QAT 4.22 driver version.

Thanks ,

Sai

[Ronny_G_Intel]

Hi sai19,

Can you please help me understand which specific cipher(s) you are observing a performance drop with. Is this is a symmetric crypto operation?

Thanks,

Ronny G

[sai19]

Yes we are right, we use only symmetric crypto. We are seeing drop in symmetric crypto throughput - around 20% drop with AES-CBC-128 and AES-CBC-256. But where as with AES-GCM-128 and AES-GCM-256 we are seeing 25% drop compared to our numbers from dpdk 20.11.2.

When we debugged we see that the qat queue is getting full and it seems that qat is taking more time to process the packets from our analysis with debug logs - not able to progress further debug beyond this till now.

Not sure if there is some tweak required for QAT gen3 with dpdk 22.11.2 or some flag need to be set, since we use qat gen2 in other platforms where we are not hitting any issue.

Thanks,
Sai

[Ronny_G_Intel]

Hi sai19,

Thanks for answering my questions, let me check with the DPDK team and I will get back to you as soon as possible.

Regards,

Ronny G

[Ronny_G_Intel]

Hi sai19,

I am trying to replicate this issue for the DPDK team to help me review it, however, when I look for the official DPDK packages for download, I cant find DPDK 20.11.2 or DPDK 22.11.2 versions, I am checking at https://core.dpdk.org/download/

Is there any particular reason why you are using these versions, especially version DPDK 22.11.2 since you upgraded from 20.11.2?

Did you try the latest stable version DPDK 23.11.0? Is there any reason why you didn't upgrade the system to this version? ...or to any other DPDK version more recent than DPDK 22.11.2?

Regards,

Ronny G

[sai19]

Hi @Ronny_G_Intel , please find the versions available under tags at https://git.dpdk.org/dpdk-stable/?h=v22.11.2

"Is there any particular reason why you are using these versions, especially version DPDK 22.11.2 since you upgraded from 20.11.2?" - To answer this, there is no specific reason for this, when we planned for upgrade this was the latest version released from intel so we went ahead with this.

We haven't tried v23.11.0/recent versions - it's quite hard to upgrade this to now, since it would involve lot of effort as well. But on that note, I have ported the changes by some commits in common/qat and crypto/qat to see if that fixes - but there was no help.

Thanks, 

-Sai

[Ronny_G_Intel]

Hi sai19,

I apologize for the delay in getting back to you.

I filed an internal case with the DPDK team, they will look into this issue and will get back to me as soon as possible and I will contact you immediately.

I hope this issue gets resolved soon.

Regards,

Ronny G

[Ronny_G_Intel]

Hi sai19,

As mentioned before, I filed and internal ticket to look into this regression issue after DPDK upgrade and we found and identical case reported in our IPS tool, I believe this community post is a duplicate of the IPS case.

Instructions regarding how to mitigate this issue were provided via IPS and plans to investigate further are in progress.

Can you please confirm that you are receiving appropriate support for this issue via IPS and if the answer is Yes, can I close this ticket?

Thanks,

Ronny G

[Ronny_G_Intel]

Hi sai19,

I hope all is well with you.

Following up on this request. Please let me know if there is anything else we can do for you or if I can close it as duplicate.

Regards,

Ronny G

[sai19]

Hi @Ronny_G_Intel ,

First of all, my apologies for late reply - was away last week so.

Yes you are correct, one of my colleague raised a IPS ticket for the same. Thank you for your coordination throughout, you can close this - we would be tracking this further with IPS ticket.

Thanks & Regards,

Sai

[Ronny_G_Intel]

Hi sai19,

Thanks for confirming, I will be closing this ticket and will monitor the IPS ticket for resolution.

Regards,

Ronny G