Your Location is: Home > Mysql

RDS MySQL Connection Timeout Over Heroku to AWS Peering Connection

From: Portugal View: 4192 Dillen 

Question

I'm seeing an intermittent connection timeout when trying to establish a connection from a Heroku one-off dyno to a MySQL RDS instance over Heroku's Private Space Peering Connection. The error is a ETIMEDOUT error using the node mysql2 library. It occurs at a frequency of about 1/100 attempts with no pattern as to time of day. It's also worth noting a few of the following:

  • This is a staging environment and has minimal traffic so the RDS metrics are mostly flat. Resource contention is likely not the issue. CPU hovers at 1%, IOPS are negligible, connection count is <10, storage and memory are over-provisioned.
  • This error only started happening when I switched to using the peering connection as opposed to connecting to the database over the public internet.
  • I do not see this error when using connection pooling on longer running processes.
  • MySQL error logs are clean

These are the potential issues I've investigated and ruled out:

  • VPC routes exist for all RDS subnets using 10.0.0.0/16 (Heroku CIDR) via the correct peering connection
  • Network ACLS applied at subnet level are ANY/ANY for all RDS subnets
  • RDS allows ingress traffic via 10.0.0.0/16 (Heroku CIDR)
  • DNS resolution works across the peering connection

The current MySQL connection configuration is using the default 10000 connection timeout. I'm hesitant to raise this without understanding the root cause because 10s should be plenty to establish a connection. Here is the code for connecting and tearing down the connection:

// Heroku dyno boots up via the scheduler add-on

const connection = await mysql.createConnection({
    timezone: '+00:00',
    dateStrings: ['DATE', 'DATETIME'],
    decimalNumbers: true,
    host: process.env.MYSQL_HOST,
    database: process.env.MYSQL_DATABASE,
    user: process.env.MYSQL_USER,
    password: process.env.MYSQL_PASSWORD,
    port: parseInt(process.env.MYSQL_PORT, 10),
    ssl: fs.readFileSync('...', 'utf8')
});

// Do some work

await connection.end();

// Heroku dyno is torn down

It's difficult to get insight into the Heroku private space side of things because it's so abstracted but has anyone seen this kind of issue?

Best answer