• Icon: Bug Bug
    • Resolution: Done
    • Icon: Major Major
    • 4.2.2b1
    • None
    • Backend
    • False
    • False
    • Undefined
    • AAH 4.3.0 Sprint 1, AAH 4.3.0 Sprint 2

      The Automation Hub team has experienced issues we believe can be traced to Akamai rate limiting. The current limit is easily triggered by a very valid feature in our product: content sync from c.rh.c to customer's on-premise systems. We need to explore any options to remove or ease this limit for our application.

            [AAH-176] Akamai rate limiting breaks Automation Hub Sync

            Update master to use pulp_ansible 0.5.3

            Adrian Likins (Inactive) added a comment - Update master to use pulp_ansible 0.5.3

            This is addressed in pulp_ansible 0.5.3. In place of the SSO token request a sleep() timeout was added. This is a temporary measure until a more robust solution is developed.

            Chris Houseknecht (Inactive) added a comment - This is addressed in pulp_ansible 0.5.3. In place of the SSO token request a sleep() timeout was added. This is a temporary measure until a more robust solution is developed.

            cspealma@redhat.com chousekn- if I am understanding this correctly, I think this ticket should go to the automation hub Jira project–do y'all mind moving to the appropriate project?

            Katie Riedesel added a comment - cspealma@redhat.com chousekn - if I am understanding this correctly, I think this ticket should go to the automation hub Jira project–do y'all mind moving to the appropriate project?

            A summary of this issue has been sent to akamai-help@redhat.com.

            Chris Houseknecht (Inactive) added a comment - A summary of this issue has been sent to akamai-help@redhat.com.

            A more specific issue, that may be a place to start, is that Akamai sends the wrong error code. We're receiving a 403 error, which indicates "access to the requested resource is forbidden". The sync machinery interprets this as a security issue, and naturally goes and gets a new SSO token. If Akamai could return a 429, which indicates "the user has sent too many requests in a given amount of time (i.e., rate limiting)", our application would know to back off and wait a period of time before continuing with the sync.

            Chris Houseknecht (Inactive) added a comment - - edited A more specific issue, that may be a place to start, is that Akamai sends the wrong error code. We're receiving a 403 error, which indicates "access to the requested resource is forbidden". The sync machinery interprets this as a security issue, and naturally goes and gets a new SSO token. If Akamai could return a 429, which indicates "the user has sent too many requests in a given amount of time (i.e., rate limiting)", our application would know to back off and wait a period of time before continuing with the sync.

            The platform infrastructure owns the Akamai configuration.  The Akamai team will do whatever we ask them once we decide a way forward.

            The rate limiting is there for security reasons (denial of service protection) , so we need to balance that vs valid use cases.

            We need to find out if its possible to customize the configuration (for example set different threshold for Automation Hub API) , but  I suggest we also look more closely at the Automation Hub sync architecture and exhaust other means of solving this problem before relaxing our WAF configuration, so we are not trading one problem for another.

             

            Lindani Phiri added a comment - The platform infrastructure owns the Akamai configuration.  The Akamai team will do whatever we ask them once we decide a way forward. The rate limiting is there for security reasons (denial of service protection) , so we need to balance that vs valid use cases. We need to find out if its possible to customize the configuration (for example set different threshold for Automation Hub API) , but  I suggest we also look more closely at the Automation Hub sync architecture and exhaust other means of solving this problem before relaxing our WAF configuration, so we are not trading one problem for another.  

            rhn-insights-aprice rhn-support-lphiri cmoore-insights

             

             Do we just need to have Calvin reach out to akamai-help@redhat.com ? Or Shawn Iwinski?

             

            Thanks!

            -kriedese

            Katie Riedesel added a comment - rhn-insights-aprice rhn-support-lphiri cmoore-insights    Do we just need to have Calvin reach out to akamai-help@redhat.com ?  Or Shawn Iwinski?   Thanks! - kriedese

              cspealma@redhat.com Clara Spealman (Inactive)
              cspealma@redhat.com Clara Spealman (Inactive)
              Votes:
              0 Vote for this issue
              Watchers:
              9 Start watching this issue

                Created:
                Updated:
                Resolved: