From e6e1302854e5ad8f5c18b01948259679e87a2e9b Mon Sep 17 00:00:00 2001 From: Ryan Kurtz Date: Mon, 24 Jun 2019 14:12:31 -0400 Subject: [PATCH] GT-2897: DevGuide reorg. --- DevGuide.md | 268 +++++++++++++++++++++++++--------------------------- 1 file changed, 127 insertions(+), 141 deletions(-) diff --git a/DevGuide.md b/DevGuide.md index 62cc99ac3f..f05b24525c 100644 --- a/DevGuide.md +++ b/DevGuide.md @@ -6,13 +6,16 @@ The following is a list of dependencies, in no particular order. This guide includes instructions for obtaining many of these at the relevant step(s). You may not need all of these, depending on which portions you are building or developing. -* JDK 11 - We test and build using OpenJDK 11.0.2. - - https://jdk.java.net/11/ +* Java JDK 11 - Free long term support (LTS) versions of JDK 11 are provided by: + - AdoptOpenJDK + - https://adoptopenjdk.net/releases.html?variant=openjdk11&jvmVariant=hotspot + - Amazon Corretto + - https://docs.aws.amazon.com/corretto/latest/corretto-11-ug/downloads-list.html * Eclipse - It must support JDK 11. Eclipse 2018-12 or later should work. Other IDEs may work, but we have not tested them. - https://www.eclipse.org/downloads/ * Gradle 5.0 - Later versions may work, but you'll need to modify our version check. - https://gradle.org/next-steps/?version=5.0&format=bin -* A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio 2017 on Windows, . +* A C/C++ compiler - We use GCC on Linux, Xcode (Clang) on macOS, and Visual Studio 2017 on Windows. - https://gcc.gnu.org/ - https://developer.apple.com/xcode/ - https://visualstudio.microsoft.com/downloads/ @@ -56,7 +59,7 @@ In Eclipse, select Window -> Prefereces (Eclipse -> Preferences on macOS), then Install Gradle, add it to your `PATH`, and ensure it is launched using JDK 11. -## Setup Source and Dependency Repositories +## Setup Source Repository You may choose any directory for your working copy, but these instructions will assume you have cloned the source to `~/git/ghidra`. Be sure to adjust the commands to match your chosen working directory if different than suggested: @@ -67,18 +70,26 @@ cd ~/git git clone git@github.com:NationalSecurityAgency/ghidra.git ``` +## Setup Build Dependency Repository + Ghidra's build uses artifacts named as available in Maven Central and Bintray JCenter, when possible. Unfortunately, in some cases, the artifact or the particular version we desire is not available. -So, in addition to mavenCentral and jcenter, you must configure a flatDir-style repository for manually-downloaded dependencies. +So, in addition to mavenCentral and jcenter, you must configure a flat directory-style repository for +manually-downloaded dependencies. -#### Dependency Setup Script -The flat repository mentioned above can be done automatically by running a simple Gradle script. -Navigate to the Ghidra clone you just created and from the top-level ghidra folder, run the following: +The flat directory-style repository can be created and populated automatically by a provided script, +or manually by downloading the required dependencies. Choose one of the two following methods: + * [Automatic script instructions](#automatic-script-instructions) + * [Manual download instructions](#manual-download_instructions) + +### Automatic Script Instructions +The flat directory-style repository can be setup automatically by running a simple Gradle script. +Navigate to the Ghidra clone directory you just created, and run the following: ``` gradle --init-script gradle/init.gradle tasks ``` The Gradle task to be executed, in this case _tasks_, is unimportant. The point is to have Gradle execute -the __init.gradle__ script. If it ran correctly you will have a new folder, __flatRepo/__, in your home directory populated with the following jar files: +the `init.gradle` script. If it ran correctly you will have a new folder, `flatRepo/`, in your home directory populated with the following jar files: * AXMLPrinter2 * csframework * dex-ir-2.0 @@ -93,7 +104,10 @@ the __init.gradle__ script. If it ran correctly you will have a new folder, __fl There will also be a new archive, yajsw-stable-12.12.zip, placed in __ghidra.bin/Ghidra/Features/GhidraServer/__. -If you see these, congrats! Skip to [importing the Gradle project](#import-gradle-project). If not, continue with manual configuration below... +If you see these, congrats! Skip to [building](#building-ghidra) or [developing](#developing-ghidra). If not, continue with manual download +instructions below... + +### Manual Download Instructions Create `~/.gradle/init.d/repos.gradle` with the following contents: @@ -118,7 +132,7 @@ mkdir ~/flatRepo If you prefer not to modify your user-wide Gradle configuration, you may use Gradle's other init script facilities, but you're on your own. -## Get Dependencies for FileFormats: +#### Get Dependencies for FileFormats: Download `dex-tools-2.0.zip` from the dex2jar project's releases page on GitHub. Unpack the `dex-*.jar` files from the `lib` directory to `~/flatRepo`: @@ -139,7 +153,7 @@ cd ~/flatRepo curl -OL https://storage.googleapis.com/google-code-archive-downloads/v2/code.google.com/android4me/AXMLPrinter2.jar ``` -## Get Dependencies for DMG: +#### Get Dependencies for DMG: Download `hfsexplorer-0_21-bin.zip` from www.catacombae.org. Unpack the `lib` directory to `~/flatRepo.`: @@ -154,11 +168,77 @@ cd lib cp csframework.jar hfsx_dmglib.jar hfsx.jar iharder-base64.jar ~/flatRepo/ ``` -## Import Gradle Project +#### Get Dependencies for GhidraServer -If you want just to build Ghidra, you may skip ahead to Building Ghidra. -Otherwise, import Ghidra into Eclipse using the integrated BuildShip plugin. -Select File -> Import, expand Gradle, and select "Existing Gradle Project." +Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12. +Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in a directory named: +`ghidra.bin/Ghidra/Features/GhidraServer/`. Note that `ghidra.bin` must be a sibling of `ghidra`: + +```bash +cd ~/Downloads # Or wherever +curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip +mkdir -p ~/git/ghidra.bin/Ghidra/Features/GhidraServer/ +cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra.bin/Ghidra/Features/GhidraServer/ +``` + +#### Get Dependencies for GhidraDev + +Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse. +Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in a directory named: +`ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/`. Note that +`ghidra.bin` must be a sibling of `ghidra`. + +```bash +cd ~/Downloads # Or wherever +curl -OL 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip' +curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'http://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip' +sha512sum -c 'cdt-8.6.0.zip.sha512' +mkdir -p ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/ +cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/ +``` + +Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory: + +```bash +cd ~/Downloads # Or wherever +curl -OL https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip +cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/ +``` + +## Building Ghidra + +Before building, you may want to update the version and release name. +These properties are kept in `Ghidra/application.properties`. + +If you want it included, you must also build the GhidraDevPlugin module first. +Some supporting data will also be missing. +See the sections below for instructions to produce these components. +You may also be able to copy some of this data from a previous official distribution. + +To build the full package, use Gradle: + +```bash +gradle buildGhidra +``` + +The output will be placed in `build/dist/`. +It will be named according to the version, release name, build date, and platform. +To test it, unzip it where you like, and execute `./ghidraRun`. + +## Developing Ghidra + +### Prepare the Environment + +From the project root, execute: + +```bash +gradle prepDev +``` +The `prepDev` tasks primarily include generating some source, indexing our built-in help, and unpacking some dependencies. + +### Import Eclipse Projects +To develop/modify Ghidra, import Ghidra into Eclipse using the integrated BuildShip plugin. +Select __File -> Import__, expand Gradle, and select "Existing Gradle Project." Select the root of the source repo as the root Gradle project. Be sure to select Gradle 5.0, or point it at your local installation. You may see build path errors until the environment is properly prepared, as described below. @@ -171,32 +251,11 @@ From the project root: gradle eclipse ``` -Select File -> Import, expand General, and select "Existing Projects into Workspace." +Select __File -> Import__, expand General, and select "Existing Projects into Workspace." Select the root of the source repo, and select "Search for nested projects." Select all, and Finish. You may see build path errors until the environment is properly prepared, as described below. -## Prepare the Environment - -From the project root, execute: - -```bash -gradle prepDev -x yajswDevUnpack eclipse -``` -The `prepDev` tasks primarily include generating some source, indexing our built-in help, and unpacking some dependencies. -Regarding `yajswDevUnpack`, please see the relevant sections on GhidraServer below. -For now, we exclude the unpack task. - -Optionally, to pre-compile all the language modules, you may also execute: - -```bash -gradle sleighCompile -``` - -Refresh the projects in Eclipse. -You should not see any errors at this point, and you can accomplish many development tasks. -However, some features of Ghidra will not be functional until further steps are taken. - ### Building the natives Some of Ghidra's components are built for the native platform. @@ -226,68 +285,46 @@ gradle buildNatives_win64 This will build the decompiler, the demangler for GNU toolchains, the sleigh compiler, and (on Windows only) the PDB parser. -## Run Ghidra from Eclipse +### Pre-compile Language Modules (optional) + +Optionally, to pre-compile all the language modules, you may also execute: + +```bash +gradle sleighCompile +``` + +If the language modules are not pre-compiled, Ghidra will compile them at run time on an as-needed basis. + +### Import GhidraDev project (optional) + +Developing the GhidraDev Eclipse plugin requires the _Eclipse PDE (Plug-in Development Environment)_, which +can be installed via the Eclipse marketplace. It is also included in the _Eclipse IDE for RCP and RAP Developers_. +To generate the GhidraDev Eclipse projects, execute: + +``` +gradle eclipse -PeclipsePDE +``` + +Import the newly generated GhidraDev projects into Eclipse. + +__Note:__ If you are getting compilation errors related to PyDev and CDT, go into Eclipse's preferences, +and under _Target Platform_, activate _/Eclipse GhidraDevPlugin/GhidraDev.target_. + +### Run/Debug Ghidra from Eclipse To run or debug Ghidra from Eclipse, use the provided launch configuration (usually under the "Run" or "Debug" buttons). If the launcher does not appear, it probably has not been marked as a favorite. Click the dropdown next to the "Run" button and select "Run Configurations." Then expand "Java Application" on the left to find the "Ghidra" launcher. +## Building Supporting Data -# Building Ghidra - -To build the full Ghidra distribution, you must also build the GhidraServer. - -## Get Dependencies for GhidraServer - -_Note_: If you already ran the [dependency setup script](#dependency-setup-script), you can skip this section and continue on to [building the package](#building-the-package). - -Building the GhidraServer requires "Yet another Java service wrapper" (yajsw) version 12.12. -Download `yajsw-stable-12.12.zip` from their project on www.sourceforge.net, and place it in a directory named: -`ghidra.bin/Ghidra/Features/GhidraServer/`. Note that `ghidra.bin` must be a sibling of `ghidra`: - -```bash -cd ~/Downloads # Or wherever -curl -OL https://sourceforge.net/projects/yajsw/files/yajsw/yajsw-stable-12.12/yajsw-stable-12.12.zip -mkdir -p ~/git/ghidra.bin/Ghidra/Features/GhidraServer/ -cp ~/Downloads/yajsw-stable-12.12.zip ~/git/ghidra.bin/Ghidra/Features/GhidraServer/ -``` - -Use Gradle to unpack the wrapper for development. -From your clone: - -```bash -gradle yajswDevUnpack -``` - -## Building the Package - -Before building, you may want to update the version and release name. -These properties are kept in `Ghidra/application.properties`. - -If you want it included, you must also build the GhidraDevPlugin module first. -Some supporting data will also be missing. -See the sections below for instructions to produce these components. -You may also be able to copy some of this data from a previous official distribution. - -To build the full package, use Gradle: - -```bash -gradle buildGhidra -``` - -The output will be placed in `build/dist/`. -It will be named according to the version, release name, build date, and platform. -To test it, unzip it where you like, and execute `./ghidraRun`. - -# Building Supporting Data - -Some features of Ghidra require the curation of rather extensive data bases. +Some features of Ghidra require the curation of rather extensive databases. These include the Data Type Archives and Function ID Databases, both of which require collecting header files and libraries for the relevant SDKs and platforms. Much of this work is done by hand. The archives included in our official builds can be found in the [ghidra-data] repository. -## Building Data Type Archives +### Building Data Type Archives This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning. From a CodeBrowser window, select File -> Parse C Source. @@ -295,7 +332,7 @@ From here you can create and configure parsing profiles, which lists headers and Then, click "Parse to File" to create the Data Type Archive. The result can be added to an installation or source tree by copying it to `Ghidra/Features/Base/data/typeinfo`. -## Building FID Databases +### Building FID Databases This task is often done manually from the Ghidra GUI, and the archives included in our official build require a fair bit of fine tuning. You will first need to import the relevant libraries from which you'd like to produce a FID database. @@ -310,54 +347,3 @@ Now, select Tools -> Function ID -> Populate FidDb from programs. Fill out the options appropriately and click OK. If you'd like some details of our fine tuning, take a look at `Ghidra/Features/FunctionID/building_fid.txt`. - -# Developing / Building the GhidraDev Plugin - -First, install the Eclipse Plugin Development Environment (PDE). -By default, the GhidraDev project is excluded from the build. -To enable it, uncomment it in `settings.gradle`. - -```bash -${EDITOR:-vi} settings.gradle -``` - -You will need some additional runtime dependencies: - -## Get Dependencies for GhidraDev - -Building the GhidraDev plugin for Eclipse requires the CDT and PyDev plugins for Eclipse. -Download `cdt-8.6.0.zip` from The Eclipse Foundation, and place it in a directory named: -`ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/`. Note that -`ghidra.bin` must be a sibling of `ghidra`. - -```bash -cd ~/Downloads # Or wherever -curl -OL 'http://www.eclipse.org/downloads/download.php?r=1&protocol=https&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip' -curl -o 'cdt-8.6.0.zip.sha512' -L --retry 3 'http://www.eclipse.org/downloads/sums.php?type=sha512&file=/tools/cdt/releases/8.6/cdt-8.6.0.zip' -sha512sum -c 'cdt-8.6.0.zip.sha512' -mkdir -p ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/ -cp ~/Downloads/cdt-8.6.0.zip ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/ -``` - -Download `PyDev 6.3.1.zip` from www.pydev.org, and place it in the same directory: - -```bash -cd ~/Downloads # Or wherever -curl -OL https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip -cp ~/Downloads/'PyDev 6.3.1.zip' ~/git/ghidra.bin/GhidraBuild/EclipsePlugins/GhidraDev/buildDependencies/ -``` - -Use Gradle to unpack the dependencies. -Note that these tasks will not work until you enable the GhidraDev project in `settings.gradle`. -From your clone: - -```bash -gradle cdtUnpack pyDevUnpack -``` - -## Import the GhidraDev Project - -If you're using BuildShip, simply refresh the Gradle project in Eclipse. -If you're not using BuildShip, re-run `gradle eclipse` and import the new project. - -[ghidra-data]: https://github.com/NationalSecurityAgency/ghidra-data