Faking Human Input While Interfacing with X11 Applications with xdotool + xwininfo
Just discovered xdotool - perfect tool for interfacing with closed source
applications via GUI. In my case I had an application whose API was closed
but provided a much needed bridge to an instant messaging network.
By utilising GUI control it was possible to automate a connection without
spending time reverse engineering the protocol.
Download here: http://www.semicomplete.com/projects/xdotool/
Basic usage:
1. Identify your target window
You can do this in a few ways.
A) Start the application, sleep a little, then query for the currently focused window using:
xdotool getactivewindow
B) Search for the window by title:
xdotool --onlyvisible --name ClosedSourceApp
Note that you can save this to an environment variable for reuse, eg: WINID=`xdotool getactivewindow`
2. Position mouse to window's top corner
To set the position of the mouse to 0,0 offset from the top-left of the window, first use
xwininfo to determine window position...
TOP_LEFT_X=`xwininfo -all -int -id $WINID|grep Absolute |grep X |cut -d ':' -f2 |sed 's/ //g'`
TOP_LEFT_Y=`xwininfo -all -int -id $WINID|grep Absolute |grep Y |cut -d ':' -f2 |sed 's/ //g'`
Now use those coordinates to position the mouse absolutely.
xdotool mousemove $TOP_LEFT_X $TOP_LEFT_Y
NOTE: With curved corner windows, clicking at this point may select another window, and de-focus
your target.
3. Gather further window information
It is usually useful to know the window's height and width.
WIDTH=`xwininfo -all -int -id $WINID|grep Width|cut -d ':' -f2 |cut -d ' ' -f2`
HEIGHT=`xwininfo -all -int -id $WINID|grep Height|cut -d ':' -f2 |cut -d ' ' -f2`
You can calculate relative positions based on percentages, eg: using the 'bc' tool on the
command line:
CENTER_X=`echo $WIDTH/2|bc`
CENTER_Y=`echo $HEIGHT/2|bc`
4. Perform relative pointer positioning
If you, for instance, wanted the pointer in the middle of the window, you would do this:
xdotool mousemove_relative $CENTER_X $CENTER_Y
If you wanted the pointer centered horizontally, but 135 pixels above the bottom of the
window:
xdotool mousemove_relative $CENTER_X `echo $HEIGHT-135|bc`
5. Perform input
For mouse input, syntax is simply 'xdotool click <button number>':
xdotool click 1
For keyboard input, you can use 'xdotool type <text>':
xdotool type 'All your closed source apps are belong to us.'
That's the basics. In order to properly deal with the complexities of some appliations,
it will be necessary to bring in some additional tools and techniques.
1. Visual layout analysis
It is possible to query out the structure of a window using xwininfo's -tree option. This
gives a full depth breakdown of each of the elements within the window, including their
sizes, absolute geometries, and location within the positioning heirarchy. For example:
----------------------------------------------------------------------------------------------
xwininfo: Window id: 54526669 "ClosedSourceApp"
Root window id: 58 (the root window) (has no name)
Parent window id: 8999209 (has no name)
22 children:
54526701 (has no name): () 66x20+62+527 +924+752
54526700 (has no name): () 22x22+84+501 +946+726
54526699 (has no name): () 22x22+62+501 +924+726
54526698 (has no name): () 46x46+6+500 +868+725
54526695 (has no name): () 227x373+2+121 +864+346
2 children:
54526697 (has no name): () 643x25+0+0 +864+346
2 children:
54526751 (has no name): () 6x25+640+0 +1504+346
54526749 (has no name): () 643x25+0+0 +864+346
54526696 (has no name): () 643x373+0+0 +864+346
54526694 (has no name): () 1x1+-1+-1 +861+224
54526693 (has no name): () 1x1+-1+-1 +861+224
54526690 (has no name): () 227x373+2+121 +864+346
2 children:
54526692 (has no name): () 227x21+0+0 +864+346
2 children:
54526748 (has no name): () 6x21+224+0 +1088+346
54526746 (has no name): () 227x21+0+0 +864+346
54526691 (has no name): () 227x373+0+0 +864+346
54526689 (has no name): () 1x1+-1+-1 +861+224
54526688 (has no name): () 1x1+-1+-1 +861+224
54526685 (has no name): () 227x373+2+121 +864+346
2 children:
54526687 (has no name): () 227x25+0+0 +864+346
6 children:
54526745 (has no name): () 6x25+224+0 +1088+346
54526743 (has no name): () 108x25+119+0 +983+346
54526742 (has no name): () 6x25+116+0 +980+346
54526740 (has no name): () 119x25+0+0 +864+346
54526739 (has no name): () 6x1+-3+0 +861+346
54526737 (has no name): () 1x1+-1+-1 +863+345
54526686 (has no name): () 227x373+0+0 +864+346
54526684 (has no name): () 1x1+-1+-1 +861+224
54526683 (has no name): () 1x1+-1+-1 +861+224
54526682 (has no name): () 227x30+2+91 +864+316
54526681 (has no name): () 22x20+206+67 +1068+292
54526680 (has no name): () 146x20+72+47 +934+272
54526679 (has no name): () 32x22+72+25 +934+250
54526678 (has no name): () 58x58+7+28 +869+253
54526677 (has no name): () 43x18+180+0 +1042+225
54526676 (has no name): () 26x18+154+0 +1016+225
54526675 (has no name): () 26x18+128+0 +990+225
54526670 (has no name): () 1x1+-1+-1 +861+224
----------------------------------------------------------------------------------------------
You will notice that the children within the tree also feature their own window IDs. The
general structure of an entry like "54526687 (has no name): () 227x25+0+0 +864+346" is
as follows:
<window id> (<name>): () <size_geometry> <absolute_geometry>
Some comments on those elements:
<name> 'has no name' or a quoted string
<size_geometry> <size_x>+<size_y>[+|-]<x_offset>[+|-]<y_offset>
<absolute_geometry> [+|-]<absolute_x_position>[+|-]<absolute_y_position>
You can parse this out and then recurse to build up knowledge of the layout logic of the
window. But first, a quick hint: you can often find the subgroup of window IDs you are
interested in getting at within the '-tree' output by simply counting elements and finding
a matching '<x> children:' entry.
To be sure you understand, examine the first child of the group of 6 children, ID 54526745.
54526745 (has no name): () 6x25+224+0 +1088+346
Apparently its absolute position is 1088,346. But why? Well, you can see that it has a
224 pixel horizontal offset ("+224"), but zero vertical offset ("+0") from its parent.
If you have a complex window with many children in the tree, you can remove part of the
output quickly using grep with the absolute geometry. For example, to remove all of the
elements at absolute Y positions from 200 to 299, you could run:
xwininfo -tree -int -id $WINID |grep -v '+2..$'
This is a rapid way to remove visual clutter and drill down to the element you want.
Note that xwininfo apparently does some guessing with borders to get geometry data, so
sometimes its output is incorrect - your milage may vary.
Faking Human Input While Interfacing with X11 Applications (xdotool + xwininfo)
===============================================================================
Just discovered xdotool - perfect tool for interfacing with closed source
applications via GUI (ie: so they think you're a proper user).
Download here: http://www.semicomplete.com/projects/xdotool/
Basic usage:
1. Identify your target window
You can do this in a few ways.
A) Start the application, sleep a little, then query for the currently focused window using:
xdotool getactivewindow
B) Search for the window by title:
xdotool --onlyvisible --name ClosedSourceApp
Note that you can save this to an environment variable for reuse, eg: WINID=`xdotool getactivewindow`
2. Position mouse to window's top corner
To set the position of the mouse to 0,0 offset from the top-left of the window, first use
xwininfo to determine window position...
TOP_LEFT_X=`xwininfo -all -int -id $WINID|grep Absolute |grep X |cut -d ':' -f2 |sed 's/ //g'`
TOP_LEFT_Y=`xwininfo -all -int -id $WINID|grep Absolute |grep Y |cut -d ':' -f2 |sed 's/ //g'`
Now use those coordinates to position the mouse absolutely.
xdotool mousemove $TOP_LEFT_X $TOP_LEFT_Y
NOTE: With curved corner windows, clicking at this point may select another window, and de-focus
your target.
3. Gather further window information
It is usually useful to know the window's height and width.
WIDTH=`xwininfo -all -int -id $WINID|grep Width|cut -d ':' -f2 |cut -d ' ' -f2`
HEIGHT=`xwininfo -all -int -id $WINID|grep Height|cut -d ':' -f2 |cut -d ' ' -f2`
You can calculate relative positions based on percentages using the 'bc' tool on the command
line, eg:
CENTER_X=`echo $WIDTH/2|bc`
CENTER_Y=`echo $HEIGHT/2|bc`
4. Perform relative pointer positioning
If you, for instance, wanted the pointer in the middle of the window, you would do this:
xdotool mousemove_relative $CENTER_X $CENTER_Y
If you wanted the pointer centered horizontally, but 135 pixels above the bottom of the
window:
xdotool mousemove_relative $CENTER_X `echo $HEIGHT-135|bc`
5. Perform input
For mouse input, syntax is simply 'xdotool click <button number>':
xdotool click 1
For keyboard input, you can use 'xdotool type <text>':
xdotool type 'All your closed source apps are belong to us.'
That's the basics. Now, in order to properly deal with the complexities of a real appliation,
it will be necessary to bring in some additional tools and techniques.
1. Visual layout analysis
It is possible to query out the structure of a window using xwininfo's -tree option. This
gives a full depth breakdown of each of the elements within the window, including their
sizes, absolute geometries, and location within the positioning heirarchy. For example:
----------------------------------------------------------------------------------------------
xwininfo: Window id: 54526669 "ClosedSourceApp"
Root window id: 58 (the root window) (has no name)
Parent window id: 8999209 (has no name)
22 children:
54526701 (has no name): () 66x20+62+527 +924+752
54526700 (has no name): () 22x22+84+501 +946+726
54526699 (has no name): () 22x22+62+501 +924+726
54526698 (has no name): () 46x46+6+500 +868+725
54526695 (has no name): () 227x373+2+121 +864+346
2 children:
54526697 (has no name): () 643x25+0+0 +864+346
2 children:
54526751 (has no name): () 6x25+640+0 +1504+346
54526749 (has no name): () 643x25+0+0 +864+346
54526696 (has no name): () 643x373+0+0 +864+346
54526694 (has no name): () 1x1+-1+-1 +861+224
54526693 (has no name): () 1x1+-1+-1 +861+224
54526690 (has no name): () 227x373+2+121 +864+346
2 children:
54526692 (has no name): () 227x21+0+0 +864+346
2 children:
54526748 (has no name): () 6x21+224+0 +1088+346
54526746 (has no name): () 227x21+0+0 +864+346
54526691 (has no name): () 227x373+0+0 +864+346
54526689 (has no name): () 1x1+-1+-1 +861+224
54526688 (has no name): () 1x1+-1+-1 +861+224
54526685 (has no name): () 227x373+2+121 +864+346
2 children:
54526687 (has no name): () 227x25+0+0 +864+346
6 children:
54526745 (has no name): () 6x25+224+0 +1088+346
54526743 (has no name): () 108x25+119+0 +983+346
54526742 (has no name): () 6x25+116+0 +980+346
54526740 (has no name): () 119x25+0+0 +864+346
54526739 (has no name): () 6x1+-3+0 +861+346
54526737 (has no name): () 1x1+-1+-1 +863+345
54526686 (has no name): () 227x373+0+0 +864+346
54526684 (has no name): () 1x1+-1+-1 +861+224
54526683 (has no name): () 1x1+-1+-1 +861+224
54526682 (has no name): () 227x30+2+91 +864+316
54526681 (has no name): () 22x20+206+67 +1068+292
54526680 (has no name): () 146x20+72+47 +934+272
54526679 (has no name): () 32x22+72+25 +934+250
54526678 (has no name): () 58x58+7+28 +869+253
54526677 (has no name): () 43x18+180+0 +1042+225
54526676 (has no name): () 26x18+154+0 +1016+225
54526675 (has no name): () 26x18+128+0 +990+225
54526670 (has no name): () 1x1+-1+-1 +861+224
----------------------------------------------------------------------------------------------
You will notice that the children within the tree also feature their own window IDs. The
general structure of an entry like "54526687 (has no name): () 227x25+0+0 +864+346" is
as follows:
<window id> (<name>): () <size_geometry> <absolute_geometry>
Some comments on those elements:
<name> 'has no name' or a quoted string
<size_geometry> <size_x>+<size_y>[+|-]<x_offset>[+|-]<y_offset>
<absolute_geometry> [+|-]<absolute_x_position>[+|-]<absolute_y_position>
You can parse this out and then recurse to build up knowledge of the layout logic of the
window. But first, a quick hint: you can often find the subgroup of window IDs you are
interested in getting at within the '-tree' output by simply counting elements and finding
a matching '<x> children:' entry.
To be sure you understand, examine the first child of the group of 6 children, ID 54526745.
54526745 (has no name): () 6x25+224+0 +1088+346
Apparently its absolute position is 1088,346. But why? Well, you can see that it has a
224 pixel horizontal offset ("+224"), but zero vertical offset ("+0") from its parent.
If you have a complex window with many children in the tree, you can remove part of the
output quickly using grep with the absolute geometry. For example, to remove all of the
elements at absolute Y positions from 200 to 299, you could run:
xwininfo -tree -int -id $WINID |grep -v '+2..$'
This is a rapid way to remove visual clutter and drill down to the element you want.
Note that xwininfo does some guessing to get geometry, and sometimes its output is
incorrect, so your milage may vary.
Root window id: 58 (the root window) (has no name)
Parent window id: 8999209 (has no name)
22 children:
54526701 (has no name): () 66x20+62+527 +924+752
54526700 (has no name): () 22x22+84+501 +946+726
54526699 (has no name): () 22x22+62+501 +924+726
54526698 (has no name): () 46x46+6+500 +868+725
54526695 (has no name): () 227x373+2+121 +864+346
2 children:
54526697 (has no name): () 643x25+0+0 +864+346
2 children:
54526751 (has no name): () 6x25+640+0 +1504+346
54526749 (has no name): () 643x25+0+0 +864+346
54526696 (has no name): () 643x373+0+0 +864+346
54526694 (has no name): () 1x1+-1+-1 +861+224
54526693 (has no name): () 1x1+-1+-1 +861+224
54526690 (has no name): () 227x373+2+121 +864+346
2 children:
54526692 (has no name): () 227x21+0+0 +864+346
2 children:
54526748 (has no name): () 6x21+224+0 +1088+346
54526746 (has no name): () 227x21+0+0 +864+346
54526691 (has no name): () 227x373+0+0 +864+346
54526689 (has no name): () 1x1+-1+-1 +861+224
54526688 (has no name): () 1x1+-1+-1 +861+224
54526685 (has no name): () 227x373+2+121 +864+346
2 children:
54526687 (has no name): () 227x25+0+0 +864+346
6 children:
54526745 (has no name): () 6x25+224+0 +1088+346
54526743 (has no name): () 108x25+119+0 +983+346
54526742 (has no name): () 6x25+116+0 +980+346
54526740 (has no name): () 119x25+0+0 +864+346
54526739 (has no name): () 6x1+-3+0 +861+346
54526737 (has no name): () 1x1+-1+-1 +863+345
54526686 (has no name): () 227x373+0+0 +864+346
54526684 (has no name): () 1x1+-1+-1 +861+224
54526683 (has no name): () 1x1+-1+-1 +861+224
54526682 (has no name): () 227x30+2+91 +864+316
54526681 (has no name): () 22x20+206+67 +1068+292
54526680 (has no name): () 146x20+72+47 +934+272
54526679 (has no name): () 32x22+72+25 +934+250
54526678 (has no name): () 58x58+7+28 +869+253
54526677 (has no name): () 43x18+180+0 +1042+225
54526676 (has no name): () 26x18+154+0 +1016+225
54526675 (has no name): () 26x18+128+0 +990+225
54526670 (has no name): () 1x1+-1+-1 +861+224
----------------------------------------------------------------------------------------------
You will notice that the children within the tree also feature their own window IDs. The
general structure of an entry like "54526687 (has no name): () 227x25+0+0 +864+346" is
as follows:
<window id> (<name>): () <size_geometry> <absolute_geometry>
Some comments on those elements:
<name> 'has no name' or a quoted string
<size_geometry> <size_x>+<size_y>[+|-]<x_offset>[+|-]<y_offset>
<absolute_geometry> [+|-]<absolute_x_position>[+|-]<absolute_y_position>
You can parse this out and then recurse to build up knowledge of the layout logic of the
window. But first, a quick hint: you can often find the subgroup of window IDs you are
interested in getting at within the '-tree' output by simply counting elements and finding
a matching '<x> children:' entry.
To be sure you understand, examine the first child of the group of 6 children, ID 54526745.
54526745 (has no name): () 6x25+224+0 +1088+346
Apparently its absolute position is 1088,346. But why? Well, you can see that it has a
224 pixel horizontal offset ("+224"), but zero vertical offset ("+0") from its parent.
If you have a complex window with many children in the tree, you can remove part of the
output quickly using grep with the absolute geometry. For example, to remove all of the
elements at absolute Y positions from 200 to 299, you could run:
xwininfo -tree -int -id $WINID |grep -v '+2..$'
This is a rapid way to remove visual clutter and drill down to the element you want.
Note that xwininfo does some guessing to get geometry, and sometimes its output is
incorrect, so your milage may vary.
Published on pratyeka.org @ http://pratyeka.org/fake-x-input/ in January 2009