# Inappropriate IOCTL

What the heck does "Errno::ENOTTY: Inappropriate ioctl for device" mean?

I recently encountered a problem that I've had in the past, so I thought I would make a post to remind (mostly myself) how I finally solved it. This helps me avoid a groundhog day situation where I continually make the same mistake and don't remember how I fixed it, since I didn't document it properly.

## The Setup

We have a Rails app that deploys using Capistrano. As we merge to our develop and main branches, we also have a GitHub actions runner that will perform deployments (assuming the build works and tests pass). So, our setup looks something like this:

flowchart LR ldm[Local Development Machine] ghr[Github Runner] stage[stage.example.com] ldm -->|push to develop|ghr ghr -->|capistrano via ssh|stage

Capistrano runs through ssh, where it connects to GitHub to download the repository. Because this is running in an automated environment, we need to use an SSH key instead of a password to authenticate (we'll see why in just a second).

The file that controls how the deployment works is pretty simple. It's contained in config/deploy/staging.rb, and looks like this:

set :stage, :staging

set :bundle_without, 'test'

set :rails_env, fetch(:staging)

set :use_sudo, false

set :deploy_to, '/var/www/example.com/stage.api.example.com'

server "stage.api.example.com", user: "deployer-bot", roles: %w{web app db}

set :branch, ENV["REVISION"] || ENV["BRANCH_NAME"] || "develop"

set :ssh_options, {
keys: %w(~/.ssh/id_rsa),
forward_agent: true,
}

Now, when I run bundle exec cap staging deploy locally, it deploys to the server stage.api.example.com and places the revision in the /var/www/example.com/stage.api.example.com directory.

When we add GitHub actions into the mix, though, things get a little dicey. My GitHub actions definition file, located in .github/workflows/staging.yml, looks like this:

name: Build, Test, and Deploy to Staging

on:
push:
branches:
- develop
# Allows you to run this workflow manually from the Actions tab
workflow_dispatch:
jobs:
build-and-test:
runs-on: ubuntu-latest
env:
DB_DATABASE: stage_db
DB_ROOT_USER: root
DB_PASSWORD: ${{ secrets.MYSQL_USER_PASSWORD }} steps: - name: Set up MySQL run: | sudo systemctl start mysql.service mysql -e 'CREATE DATABASE${{ env.DB_DATABASE }};' -u${{ env.DB_ROOT_USER }} -p${{ env.DB_ROOT_PASSWORD }}
mysql -e "CREATE USER '${{ env.DB_USER }}'@'localhost' IDENTIFIED BY '${{ env.DB_PASSWORD }}';" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }}
mysql -e "CREATE DATABASE IF NOT EXISTS ${{ env.DB_DATABASE }};" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }} mysql -e "GRANT ALL PRIVILEGES ON${{ env.DB_DATABASE }}.* to '${{ env.DB_USER }}'@'localhost';" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }} mysql -e "FLUSH PRIVILEGES;" -u${{env.DB_ROOT_USER}} -p${{ env.DB_ROOT_PASSWORD }} - name: Install SSH key to Server uses: shimataro/ssh-key-action@v2 with: key:${{ secrets.STAGE_API_DEPLOY_KEY }}
name: github-actions
known_hosts: ${{ secrets.STAGE_API_HOST_KEY }} config: | host stage.api.example.com IdentityFile ~/.ssh/github-actions IdentitiesOnly yes ForwardAgent yes - name: Verify SSH Key run: cat ~/.ssh/github-actions - uses: actions/checkout@v2 - name: Set up Ruby Environment uses: ruby/setup-ruby@v1 with: ruby-version: 2.6.1 # runs 'bundle install' and caches installed gems automatically bundler-cache: true env: RAILS_ENV: staging - name: Setup Database env: RAILS_ENV: staging run: bundle exec rake db:setup - name: Perform Database Migrations env: RAILS_ENV: staging run: bundle exec rake db:migrate - name: Run specs env: RAILS_ENV: staging run: bundle exec rails spec deploy-staging: needs: build-and-test runs-on: ubuntu-latest steps: - name: Install SSH Host Key uses: shimataro/ssh-key-action@v2 with: key:${{ secrets.STAGE_API_DEPLOY_KEY }}
name: github-actions
known_hosts: ${{ secrets.STAGE_API_HOST_KEY }} config: | Host stage.api.example.com IdentityFile ~/.ssh/github-actions IdentitiesOnly yes ForwardAgent yes - uses: actions/checkout@v2 - name: Set up Ruby uses: ruby/setup-ruby@v1 with: # # NOTE: This is not needed since we have a .ruby-version file. # # ruby-version: 2.6.1 bundler-cache: true # runs 'bundle install' and caches installed gems automatically - name: Install SSH Key run: | eval "$(ssh-agent -s)"
- name: Check SSH Key Viability
run: |
echo "ls -al" | ssh deployer-bot@stage.api.example.com
- name: Deploy to staging
run: |
eval "\$(ssh-agent -s)"
bundle exec cap staging deploy

There's a lot going on here, but, essentially, we're setting up an SSH key to the stage.api.example.com server (this becomes important in a second), checking the viability of that SSH key, setting up ruby, and finally deploying using Capistrano. The "Check SSH Key Viability" step is unnecessary now, but it was useful in debugging the underlying issue (up next).

## The Problem

Now, even though I was able to deploy to staging locally, when it attempted to deploy on Github actions, the following error was displayed during the Deploy to staging phase:

Run bundle exec cap staging deploy
#<Thread:0x0000560949aab9c0@/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:10 run> terminated with exception (report_on_exception is true):
/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:15:in rescue in block (2 levels) in execute': Exception while executing as deployer-bot@stage.api.example.com: Inappropriate ioctl for device (SSHKit::Runner::ExecuteError)
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:11:in block (2 levels) in execute'
/home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/prompt.rb:44:in noecho': Inappropriate ioctl for device (Errno::ENOTTY)
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/methods/password.rb:68:in ask_password'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:87:in block in authenticate'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:71:in each'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh/authentication/session.rb:71:in authenticate'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/net-ssh-7.0.1/lib/net/ssh.rb:254:in start'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in call'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/connection_pool.rb:63:in with'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:177:in with_ssh'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/netssh.rb:130:in execute_command'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in block in create_command_and_execute'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in tap'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:148:in create_command_and_execute'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:61:in test'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/capistrano-passenger-0.2.1/lib/capistrano/tasks/passenger.cap:43:in block (3 levels) in <top (required)>'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in instance_exec'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/backends/abstract.rb:31:in run'
from /home/runner/work/api/api/vendor/bundle/ruby/2.6.0/gems/sshkit-1.21.3/lib/sshkit/runners/parallel.rb:12:in block (2 levels) in execute'
cap aborted!
SSHKit::Runner::ExecuteError: Exception while executing as deployer-bot@stage.api.example.com: Inappropriate ioctl for device

Caused by:
Errno::ENOTTY: Inappropriate ioctl for device

Tasks: TOP => rvm:hook => passenger:rvm:hook => passenger:test_which_passenger
(See full trace by running task with --trace)
Error: Process completed with exit code 1.

I spent quite a bit of time trying to figure out what Inappropriate ioctl for device means. I'll save you some time. It means: I require input from a terminal and no terminal is attached to this device. Had I seen the line near the bottom that says deployer-bot@stage.api.example.com's password:, I probably would've solved this a little faster, but I didn't see that because I was too busy looking through the stack trace. To be fair to me, this is a bit of a red herring, anyway, because the real error should've said something akin to git@github.com:FoamFactory/my-repo's password:.

## The Solution

What was happening was that on the GitHub runner, the SSH command to stage.api.example.com was working fine. What wasn't working, though, was the step where Capistrano, on stage.api.example.com, was checking out the codebase from GitHub. This is because the public key for deployer-bot@stage.api.example.com was not registered with GitHub under my user account, nor was there a deploy key for it for the project in question.

So, it seems as though all we need to do is add a deploy key on GitHub for the appropriate repository with the contents of the file id_rsa.pub in the .ssh directory for deployer-bot on stage.api.example.com, right? Wrong.

To complicate matters, on stage.api.example.com, the file ~/.ssh/config for deployer-bot looked like this:

Host github.com
HostName github.com
IdentityFile ~/.ssh/github-actions

What this means is that for every host except GitHub, it was using the id_rsa file located in ~/.ssh. For GitHub, though, it was using github-actions. This means that I need to add a deploy key for github-actions.pub in the same directory. Believe it or not, this still wasn't the complete answer.

Capistrano also needed to be told to use github-actions. Specifically, this line in the config/deploy/staging.rb file:

  keys: %w(~/.ssh/id_rsa),

needed to be changed to this:

  keys: %w(/.ssh/github-actions),

After that, magically, everything seemed to magically work.

## Conclusion

I hope documenting this helps you in the future. Quite honestly, I think it's a pretty specific case of misconfiguration on my side (I know this because I also posted this on StackOverflow and didn't get a response in 6 months before I figured it out on my own). That said, documenting it here will at least help me to remember what was going on, and, maybe, just maybe help someone else lost in a sea of SSH redirections.