Inappropriate IOCTL

I recently encountered a problem that I've had in the past, so I thought I would make a post to remind (mostly myself) how I finally solved it. This helps me avoid a groundhog day situation where I continually make the same mistake and don't remember how I fixed it, since I didn't document it properly.

The Setup

We have a Rails app that deploys using Capistrano. As we merge to our develop and main branches, we also have a GitHub actions runner that will perform deployments (assuming the build works and tests pass). So, our setup looks something like this:

Capistrano runs through ssh, where it connects to GitHub to download the repository. Because this is running in an automated environment, we need to use an SSH key instead of a password to authenticate (we'll see why in just a second).

The file that controls how the deployment works is pretty simple. It's contained in config/deploy/staging.rb, and looks like this:

set :stage, :staging

set :bundle_without, 'test'

set :rails_env, fetch(:staging)

set :use_sudo, false

set :deploy_to, '/var/www/example.com/stage.api.example.com'

server "stage.api.example.com", user: "deployer-bot", roles: %w{web app db}

set :branch, ENV["REVISION"] || ENV["BRANCH_NAME"] || "develop"

set :ssh_options, {
  keys: %w(~/.ssh/id_rsa),
  forward_agent: true,
}

Now, when I run bundle exec cap staging deploy locally, it deploys to the server stage.api.example.com and places the revision in the /var/www/example.com/stage.api.example.com directory.

When we add GitHub actions into the mix, though, things get a little dicey. My GitHub actions definition file, located in .github/workflows/staging.yml, looks like this:

name: Build, Test, and Deploy to Staging

on:
  push:
    branches:
      - develop
  # Allows you to run this workflow manually from the Actions tab
  workflow_dispatch:
jobs:
  build-and-test:
    runs-on: ubuntu-latest
    env:
      DB_DATABASE: stage_db
      DB_ROOT_USER: root
      DB_ROOT_PASSWORD: root
      DB_USER: eample_admin
      DB_PASSWORD: ${{ secrets.MYSQL_USER_PASSWORD }}
    steps:
      - name: Set up MySQL
        run: |
          sudo systemctl start mysql.service
          mysql -e 'CREATE DATABASE ${{ env.DB_DATABASE }};' -u${{ env.DB_ROOT_USER }} -p${{ env.DB_ROOT_PASSWORD }}
          mysql -e "CREATE USER '${{ env.DB_USER }}'@'localhost' IDENTIFIED BY '${{ env.DB_PASSWORD }}';" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
          mysql -e "CREATE DATABASE IF NOT EXISTS ${{ env.DB_DATABASE }};" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
          mysql -e "GRANT ALL PRIVILEGES ON ${{ env.DB_DATABASE }}.* to '${{ env.DB_USER }}'@'localhost';" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
          mysql -e "FLUSH PRIVILEGES;" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
      - name: Install SSH key to Server
        uses: shimataro/ssh-key-action@v2
        with:
          key: ${{ secrets.STAGE_API_DEPLOY_KEY }}
          name: github-actions
          known_hosts: ${{ secrets.STAGE_API_HOST_KEY }}
          config: |
            host stage.api.example.com
            IdentityFile ~/.ssh/github-actions
            IdentitiesOnly yes
            ForwardAgent yes
      - name: Verify SSH Key
        run: cat ~/.ssh/github-actions
      - uses: actions/checkout@v2
      - name: Set up Ruby Environment
        uses: ruby/setup-ruby@v1
        with:
          ruby-version: 2.6.1
          # runs 'bundle install' and caches installed gems automatically
          bundler-cache: true
        env:
          RAILS_ENV: staging
      - name: Setup Database
        env:
          RAILS_ENV: staging
        run: bundle exec rake db:setup
      - name: Perform Database Migrations
        env:
          RAILS_ENV: staging
        run: bundle exec rake db:migrate
      - name: Run specs
        env:
          RAILS_ENV: staging
        run: bundle exec rails spec
  deploy-staging:
    needs: build-and-test
    runs-on: ubuntu-latest
    steps:
      - name: Install SSH Host Key
        uses: shimataro/ssh-key-action@v2
        with:
          key: ${{ secrets.STAGE_API_DEPLOY_KEY }}
          name: github-actions
          known_hosts: ${{ secrets.STAGE_API_HOST_KEY }}
          config: |
            Host stage.api.example.com
            IdentityFile ~/.ssh/github-actions
            IdentitiesOnly yes
            ForwardAgent yes
      - uses: actions/checkout@v2
      - name: Set up Ruby
        uses: ruby/setup-ruby@v1
        with:
          #          # NOTE: This is not needed since we have a .ruby-version file.
          #          # ruby-version: 2.6.1
          bundler-cache: true # runs 'bundle install' and caches installed gems automatically
      - name: Install SSH Key
        run: |
          eval "$(ssh-agent -s)"
          ssh-add -D
          ssh-add ~/.ssh/github-actions
      - name: Check SSH Key Viability
        run: |
          echo "ls -al" | ssh deployer-bot@stage.api.example.com
      - name: Deploy to staging
        run: |
          eval "$(ssh-agent -s)"
          ssh-add -D
          ssh-add ~/.ssh/github-actions
          bundle exec cap staging deploy

There's a lot going on here, but, essentially, we're setting up an SSH key to the stage.api.example.com server (this becomes important in a second), checking the viability of that SSH key, setting up ruby, and finally deploying using Capistrano. The "Check SSH Key Viability" step is unnecessary now, but it was useful in debugging the underlying issue (up next).

The Problem

Now, even though I was able to deploy to staging locally, when it attempted to deploy on Github actions, the following error was displayed during the Deploy to staging phase:

Run bundle exec cap staging deploy
#<Thread:0x0000560949aab9c0@/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:15:in `rescue in block (2 levels) in execute': Exception while executing as deployer-bot@stage.api.example.com: Inappropriate ioctl for device (SSHKit::Runner::ExecuteError)
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:11:in `block (2 levels) in execute'
/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/prompt.rb:44:in `noecho': Inappropriate ioctl for device (Errno::ENOTTY)
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/prompt.rb:44:in `ask'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/methods/password.rb:68:in `ask_password'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/methods/password.rb:20:in `authenticate'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:87:in `block in authenticate'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:71:in `each'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:71:in `authenticate'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh.rb:254:in `start'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in `call'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in `with'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:177:in `with_ssh'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:130:in `execute_command'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `block in create_command_and_execute'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `tap'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in `create_command_and_execute'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:61:in `test'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/capistrano-passenger-0.2.1/lib/capistrano/tasks/passenger.cap:43:in `block (3 levels) in <top (required)>'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in `instance_exec'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in `run'
    from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:12:in `block (2 levels) in execute'
(Backtrace restricted to imported tasks)
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deployer-bot@stage.api.example.com: Inappropriate ioctl for device


Caused by:
Errno::ENOTTY: Inappropriate ioctl for device

Tasks: TOP => rvm:hook => passenger:rvm:hook => passenger:test_which_passenger
(See full trace by running task with --trace)
deployer-bot@stage.api.example.com's password:
Error: Process completed with exit code 1.

I spent quite a bit of time trying to figure out what Inappropriate ioctl for device means. I'll save you some time. It means: I require input from a terminal and no terminal is attached to this device. Had I seen the line near the bottom that says deployer-bot@stage.api.example.com's password:, I probably would've solved this a little faster, but I didn't see that because I was too busy looking through the stack trace. To be fair to me, this is a bit of a red herring, anyway, because the real error should've said something akin to git@github.com:FoamFactory/my-repo's password:.

The Solution

What was happening was that on the GitHub runner, the SSH command to stage.api.example.com was working fine. What wasn't working, though, was the step where Capistrano, on stage.api.example.com, was checking out the codebase from GitHub. This is because the public key for deployer-bot@stage.api.example.com was not registered with GitHub under my user account, nor was there a deploy key for it for the project in question.

So, it seems as though all we need to do is add a deploy key on GitHub for the appropriate repository with the contents of the file id_rsa.pub in the .ssh directory for deployer-bot on stage.api.example.com, right? Wrong.

To complicate matters, on stage.api.example.com, the file ~/.ssh/config for deployer-bot looked like this:

Host github.com
 HostName github.com
 IdentityFile ~/.ssh/github-actions

What this means is that for every host except GitHub, it was using the id_rsa file located in ~/.ssh. For GitHub, though, it was using github-actions. This means that I need to add a deploy key for github-actions.pub in the same directory. Believe it or not, this still wasn't the complete answer.

Capistrano also needed to be told to use github-actions. Specifically, this line in the config/deploy/staging.rb file:

  keys: %w(~/.ssh/id_rsa),

needed to be changed to this:

  keys: %w(/.ssh/github-actions),

After that, magically, everything seemed to magically work.

Conclusion

I hope documenting this helps you in the future. Quite honestly, I think it's a pretty specific case of misconfiguration on my side (I know this because I also posted this on StackOverflow and didn't get a response in 6 months before I figured it out on my own). That said, documenting it here will at least help me to remember what was going on, and, maybe, just maybe help someone else lost in a sea of SSH redirections.