diff --git a/Ghidra/Debug/Debugger-agent-frida/certification.manifest b/Ghidra/Debug/Debugger-agent-frida/certification.manifest index b9c2c0cdad..5f713628fb 100644 --- a/Ghidra/Debug/Debugger-agent-frida/certification.manifest +++ b/Ghidra/Debug/Debugger-agent-frida/certification.manifest @@ -1,8 +1,6 @@ ##VERSION: 2.0 ##MODULE IP: Apache License 2.0 ##MODULE IP: Apache License 2.0 with LLVM Exceptions -.classpath||NONE||reviewed||END| -.project||NONE||reviewed||END| FridaNotes.txt||GHIDRA||||END| Module.manifest||GHIDRA||||END| build.gradle||GHIDRA||||END| diff --git a/Ghidra/Debug/Debugger-agent-lldb/certification.manifest b/Ghidra/Debug/Debugger-agent-lldb/certification.manifest index e68c49c255..19e8376a8d 100644 --- a/Ghidra/Debug/Debugger-agent-lldb/certification.manifest +++ b/Ghidra/Debug/Debugger-agent-lldb/certification.manifest @@ -1,8 +1,6 @@ ##VERSION: 2.0 ##MODULE IP: Apache License 2.0 ##MODULE IP: Apache License 2.0 with LLVM Exceptions -.classpath||NONE||reviewed||END| -.project||NONE||reviewed||END| Module.manifest||GHIDRA||||END| build.gradle||GHIDRA||||END| src/llvm-project/lldb/bindings/java/java-typemaps.swig||Apache License 2.0 with LLVM Exceptions||||END| diff --git a/Ghidra/Debug/Debugger-swig-lldb/certification.manifest b/Ghidra/Debug/Debugger-swig-lldb/certification.manifest index 780959035e..fc0dff555c 100644 --- a/Ghidra/Debug/Debugger-swig-lldb/certification.manifest +++ b/Ghidra/Debug/Debugger-swig-lldb/certification.manifest @@ -1,8 +1,6 @@ ##VERSION: 2.0 ##MODULE IP: Apache License 2.0 ##MODULE IP: Apache License 2.0 with LLVM Exceptions -.classpath||NONE||reviewed||END| -.project||NONE||reviewed||END| InstructionsForBuildingLLDBInterface.txt||GHIDRA||||END| Module.manifest||GHIDRA||||END| build.gradle||GHIDRA||||END| diff --git a/Ghidra/Extensions/BSimElasticPlugin/INSTALL.txt b/Ghidra/Extensions/BSimElasticPlugin/INSTALL.txt new file mode 100755 index 0000000000..5bac4cab13 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/INSTALL.txt @@ -0,0 +1,81 @@ +Installation of the Elasticsearch BSim Plug-in: + +In order to use Elasticsearch as the back-end database for a BSim instance, +the lsh plug-in, included with this Ghidra extension, must be installed on +the Elasticsearch cluster. + +The lsh plug-in is bundled in the standard plug-in format as the file +'lsh.zip'. It must be installed separately on EVERY node of the cluster, +and each node must be restarted after the install in order for the plug-in to +become active. + +For a single node, installation is accomplished with the command-line +'elasticsearch-plugin' script that comes with the standard Elasticsearch +distribution. It expects a URL pointing to the plug-in to be installed. +The basic command, executed in the Elasticsearch installation directory +for the node, is + + bin/elasticsearch-plugin install file:///path/to/ghidra/Ghidra/Extensions/BSimElasticPlugin/data/lsh.zip + +Replace the initial portion of the absolute path in the URL to point to your +particular Ghidra installation. + +Deployment: + +Follow the Elasticsearch documentation to do any additional configuration, +starting, stopping, and management of your Elasticsearch cluster. + +To try BSim with a toy deployment, you can start a single node (as per the +documentation) from the command-line by just running + + bin/elasticsearch + +This will dump logging messages to the console, and you should see '[lsh]' +listed among the loaded plug-ins as the node starts up. + +Once the Elasticsearch node(s) are running, whether they are a toy or a full +deployment, you can immediately proceed to the BSim 'bsim' command. +The Ghidra/BSim client and 'bsim' command automatically assume an +Elasticsearch server when they see the 'https' protocol in the provided URLs, +although the 'elastic" protocol may also be specified and is equivalent. +The use of the 'http' protocol for Elasticsearch is not supported. +Adjust the hostname, port number, and repository name as appropriate. +Use a command-line similar to the following to create a BSim instance: + + bsim createdatabase elastic://1.2.3.4:9200/repo medium_32 + +This is equivalent to: + + bsim createdatabase https://1.2.3.4:9200/repo medium_32 + +Use a command-line like this to generate and commit signatures from a Ghidra Server +repository to the Elasticsearch database created above: + + bsim generatesigs ghidra://1.2.3.4/repo bsim=elastic://1.2.3.4:9200/repo + +Within Ghidra's BSim client, enter the same URL into the database connection +panel in order to place queries to your Elasticsearch deployment. See the BSim +documentation included with Ghidra for full details. + + +Version: + +The current BSim plug-in was designed and tested with Elasticsearch version 7.17.4. +A change to the Elasticsearch scripting interface, starting with version 7.15, makes the BSim +plug-in incompatible with previous versions, but the lsh plug-in jars may work without change +across later Elasticsearch versions. + +Elasticsearch plug-ins explicitly encode the version of Elasticsearch they work with, and the +plug-in script will refuse to install the lsh plug-in if its version does not match your +particular installation. If your Elasticsearch version is slightly different, you can try +unpacking the zip file, changing the version number to match your software, and then repacking +the zip file. Within the zip archive, the version number is stored in a configuration file + + elasticsearch/plugin-descriptor.properties + +The file format is fairly simple: edit the line + + elasticsearch.version=7.17.4 + +The plugin may work with other nearby versions, but proceed at your own risk. + diff --git a/Ghidra/Extensions/BSimElasticPlugin/Module.manifest b/Ghidra/Extensions/BSimElasticPlugin/Module.manifest new file mode 100755 index 0000000000..e69de29bb2 diff --git a/Ghidra/Extensions/BSimElasticPlugin/build.gradle b/Ghidra/Extensions/BSimElasticPlugin/build.gradle new file mode 100755 index 0000000000..fcce917d5d --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/build.gradle @@ -0,0 +1,99 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +apply from: "$rootProject.projectDir/gradle/distributableGhidraExtension.gradle" +apply from: "$rootProject.projectDir/gradle/javaProject.gradle" +apply plugin: 'eclipse' +eclipse.project.name = 'Xtra BSimElasticPlugin' +// This module is very different from other Ghidra modules. It is creating a stand-alone jar +// file for an elastic database plugin. It is copying files from other modules into this module +// before building a jar file from the files in this module and the cherry-picked files from +// other modules (This is very brittle and will break if any of the files are renamed or moved.) +project.ext.includeExtensionInInstallation = true + +apply plugin: 'java' + +sourceSets { + elasticPlugin { + java { + srcDirs = [ 'src', 'srcdummy', 'build/genericSrc', 'build/utilitySrc', 'build/bsimSrc' ] + } + } +} +// this dependency block is needed for this code to compile in our eclipse environment. It is not needed +// for the gradle build +dependencies { + + implementation project(':BSim') +} +libsDirName='ziplayout' + +task copyGenericTask(type: Copy) { + from project(':Generic').file('src/main/java') + into 'build/genericSrc' + include 'generic/lsh/vector/*.java' + include 'generic/hash/SimpleCRC32.java' + include 'ghidra/util/xml/SpecXmlUtils.java' +} + +task copyUtilityTask(type: Copy) { + from project(':Utility').file('src/main/java') + into 'build/utilitySrc' + include 'ghidra/xml/XmlPullParser.java' + include 'ghidra/xml/XmlElement.java' +} + +task copyBSimTask(type: Copy) { + from project(':BSim').file('src/main/java') + into 'build/bsimSrc' + include 'ghidra/features/bsim/query/elastic/ElasticUtilities.java' + include 'ghidra/features/bsim/query/elastic/Base64Lite.java' + include 'ghidra/features/bsim/query/elastic/Base64VectorFactory.java' +} + +task copyPropertiesFile(type: Copy) { + from 'contribZipExclude/plugin-descriptor.properties' + into 'build/ziplayout' +} + +task elasticPluginJar(type: Jar) { + from sourceSets.elasticPlugin.output + archiveBaseName = 'lsh' + excludes = [ + '**/org/apache', + '**/org/elasticsearch/common', + '**/org/elasticsearch/env', + '**/org/elasticsearch/index', + '**/org/elasticsearch/indices', + '**/org/elasticsearch/plugins', + '**/org/elasticsearch/script', + '**/org/elasticsearch/search' + ] +} + +task elasticPluginZip(type: Zip) { + from 'build/ziplayout' + archiveBaseName = 'lsh' + destinationDirectory = file("build/data") +} + +compileElasticPluginJava.dependsOn copyGenericTask +compileElasticPluginJava.dependsOn copyUtilityTask +compileElasticPluginJava.dependsOn copyBSimTask + +elasticPluginZip.dependsOn elasticPluginJar +elasticPluginZip.dependsOn copyPropertiesFile + +jar.dependsOn elasticPluginZip diff --git a/Ghidra/Extensions/BSimElasticPlugin/certification.manifest b/Ghidra/Extensions/BSimElasticPlugin/certification.manifest new file mode 100755 index 0000000000..3724510928 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/certification.manifest @@ -0,0 +1,6 @@ +##VERSION: 2.0 +##MODULE IP: Apache License 2.0 +INSTALL.txt||GHIDRA||||END| +Module.manifest||GHIDRA||reviewed||END| +contribZipExclude/plugin-descriptor.properties||GHIDRA||||END| +extension.properties||GHIDRA||||END| diff --git a/Ghidra/Extensions/BSimElasticPlugin/contribZipExclude/plugin-descriptor.properties b/Ghidra/Extensions/BSimElasticPlugin/contribZipExclude/plugin-descriptor.properties new file mode 100755 index 0000000000..bb6c329c67 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/contribZipExclude/plugin-descriptor.properties @@ -0,0 +1,6 @@ +description=Feature Vector Plugin +version=1.0 +name=lsh +classname=org.elasticsearch.plugin.analysis.lsh.AnalysisLSHPlugin +java.version=1.11 +elasticsearch.version=8.8.1 diff --git a/Ghidra/Extensions/BSimElasticPlugin/extension.properties b/Ghidra/Extensions/BSimElasticPlugin/extension.properties new file mode 100755 index 0000000000..5ab4262499 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/extension.properties @@ -0,0 +1,5 @@ +name=BSimElasticPlugin +description=Elastic search backend for BSim. +author=Ghidra Team +createdOn=11/23/20 +version=@extversion@ \ No newline at end of file diff --git a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/AnalysisLSHPlugin.java b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/AnalysisLSHPlugin.java new file mode 100755 index 0000000000..14cf17c1d6 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/AnalysisLSHPlugin.java @@ -0,0 +1,134 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugin.analysis.lsh; + +import java.io.IOException; +import java.util.*; + +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.env.Environment; +import org.elasticsearch.index.IndexModule; +import org.elasticsearch.index.IndexSettings; +import org.elasticsearch.index.analysis.TokenizerFactory; +import org.elasticsearch.indices.analysis.AnalysisModule.AnalysisProvider; +import org.elasticsearch.plugins.*; +import org.elasticsearch.script.ScriptContext; +import org.elasticsearch.script.ScriptEngine; + +import generic.lsh.vector.IDFLookup; +import generic.lsh.vector.WeightFactory; +import ghidra.features.bsim.query.elastic.Base64VectorFactory; +import ghidra.features.bsim.query.elastic.ElasticUtilities; + +public class AnalysisLSHPlugin extends Plugin implements AnalysisPlugin, ScriptPlugin { + + public static final String TOKENIZER_SETTINGS_BASE = "index.analysis.tokenizer.lsh_"; + public static String settingString = ""; + + static private Map vecFactoryMap = new HashMap<>(); + private Map> tokFactoryMap; + + public class TokenizerFactoryProvider implements AnalysisProvider { + + @Override + public TokenizerFactory get(IndexSettings indexSettings, Environment env, String name, + Settings settings) throws IOException { +// settingString = settingString + " : " + indexSettings.getIndex().getName() + '(' + name + ')'; + return new LSHTokenizerFactory(indexSettings, env, name, settings); + } + } + + public AnalysisLSHPlugin() { + TokenizerFactoryProvider provider = new TokenizerFactoryProvider(); + tokFactoryMap = Collections.singletonMap("lsh_tokenizer", provider); + } + + private static void setupVectorFactory(String name, String idfConfig, String lshWeights) { + WeightFactory weightFactory = new WeightFactory(); + String[] split = lshWeights.split(" "); + double[] weightArray = new double[split.length]; + for (int i = 0; i < weightArray.length; ++i) { + weightArray[i] = Double.parseDouble(split[i]); + } + weightFactory.set(weightArray); + IDFLookup idfLookup = new IDFLookup(); + split = idfConfig.split(" "); + int[] intArray = new int[split.length]; + for (int i = 0; i < intArray.length; ++i) { + intArray[i] = Integer.parseInt(split[i]); + } + idfLookup.set(intArray); + Base64VectorFactory vectorFactory = new Base64VectorFactory(); + // Server-side factory is never used to generate signatures, + // so we don't need to specify settings + vectorFactory.set(weightFactory, idfLookup, 0); + vecFactoryMap.put(name, vectorFactory); + } + + /** + * Entry point for Tokenizer and Script factories to grab the global vector factory + * @param name is the name of the tokenizer + * @return the vector factory used by the tokenizer + */ + public static Base64VectorFactory getVectorFactory(String name) { + return vecFactoryMap.get(name); + } + + @Override + public void onIndexModule(IndexModule indexModule) { + super.onIndexModule(indexModule); + + Settings settings = indexModule.getSettings(); + String name = null; + // Look for the specific kind of tokenizer settings, within the global settings for the index + for (String key : settings.keySet()) { + if (key.startsWith(TOKENIZER_SETTINGS_BASE)) { + // We can have different settings for different indices, distinguished by this name + int pos = key.indexOf('.', TOKENIZER_SETTINGS_BASE.length() + 1); + if (pos > 0) { + name = key.substring(TOKENIZER_SETTINGS_BASE.length(), pos); + break; + } + } + } + if (name != null) { + String tokenizerName = "lsh_" + name; + if (getVectorFactory(tokenizerName) != null) { + return; // Factory already exists + } + settingString = settingString + " : onModule(" + name + ')'; + // If we found LSH tokenizer settings, pull them out and construct an LSHVectorFactory with them + String baseKey = TOKENIZER_SETTINGS_BASE + name + '.'; + String idfConfig = settings.get(baseKey + ElasticUtilities.IDF_CONFIG); + String lshWeights = settings.get(baseKey + ElasticUtilities.LSH_WEIGHTS); + if (idfConfig == null || lshWeights == null) { + return; // IDF_CONFIG and LSH_WEIGHTS settings must be present to proceed + } + setupVectorFactory(tokenizerName, idfConfig, lshWeights); + } + } + + @Override + public ScriptEngine getScriptEngine(Settings settings, Collection> contexts) { + return new BSimScriptEngine(); + } + + @Override + public Map> getTokenizers() { + return tokFactoryMap; + } + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/BSimScriptEngine.java b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/BSimScriptEngine.java new file mode 100755 index 0000000000..bbb7de5aba --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/BSimScriptEngine.java @@ -0,0 +1,54 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugin.analysis.lsh; + +import java.util.*; + +import org.elasticsearch.script.*; + +public class BSimScriptEngine implements ScriptEngine { + private final static String ENGINE_NAME = "bsim_scripts"; + + @Override + public FactoryType compile(String scriptName, String scriptSource, + ScriptContext context, Map params) { + if (context.equals(ScoreScript.CONTEXT) == false) { + throw new IllegalArgumentException( + getType() + "scripts cannot be used for context [" + context.name + "]"); + } + if (VectorCompareScriptFactory.SCRIPT_NAME.equals(scriptSource)) { + ScoreScript.Factory factory = new VectorCompareScriptFactory(); + return context.factoryClazz.cast(factory); + } + throw new IllegalArgumentException("Unknown script name " + scriptSource); + } + + @Override + public void close() { + // Can free up resources + } + + @Override + public Set> getSupportedContexts() { + return Collections.singleton(ScoreScript.CONTEXT); + } + + @Override + public String getType() { + return ENGINE_NAME; + } + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHBinner.java b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHBinner.java new file mode 100755 index 0000000000..52d55e5d35 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHBinner.java @@ -0,0 +1,293 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugin.analysis.lsh; + +import generic.lsh.vector.HashEntry; +import ghidra.features.bsim.query.elastic.Base64Lite; + +/** + * Class for calculating the bin ids on LSHVectors as part of the LSH indexing process + * + */ +public class LSHBinner { + + private static final char[] hashSignTable = new char[512]; + private static int VEC_SIZE_UPPER = 5; // Size above which to use FFT to calculate dotproduct family + private static int LSH_HASHBASE = 0xd7e6a299; + private static int HASH_MULTIPLIER = 1103515245; + private static int HASH_ADDEND = 12345; + + public static class BytesRef { + public char[] buffer; + public BytesRef(int size) { buffer = new char[size]; } + } + + private int k; // Number of bits per bin id + private int L; // Number of binnings + private double doubleBuffer[]; // Scratch space for dot-product calculation + private BytesRef tokenList[]; // Final token list used by lucene + + static { + /** + * This is a precalculated table for generating dot-products with the random family of vectors directly + * The first vector r_0 is expressed as a hashing function on the dimension index and the other vectors + * are derived from r_0 using an FFT. The table is formed by precalculating the FFT on basis vectors in this table + */ + int i, j; + int[] arr = new int[16]; + int hibit0ptr; + int hibit1ptr; + + for (i = 0; i < 16; ++i) { /* For each 4-bit position */ + hibit0ptr = i * 16; + hibit1ptr = (i + 16) * 16; + for (j = 0; j < 16; ++j) + arr[j] = 0; + + arr[i] = 1; + hashFft16(arr); + for (j = 0; j < 16; ++j) { + if (arr[j] > 0) { + hashSignTable[hibit0ptr + j] = '+'; + hashSignTable[hibit1ptr + j] = '-'; + } else { + hashSignTable[hibit0ptr + j] = '-'; + hashSignTable[hibit1ptr + j] = '+'; + } + } + } + } + + /** + * Raw Fast Fourier Transform on 16 wide integer array + * @param arr is the 16-long array + */ + private static void hashFft16(int[] arr) { + int x,y; + + x = arr[0]; y = arr[8]; arr[0] = x + y; arr[8] = x - y; + x = arr[1]; y = arr[9]; arr[1] = x + y; arr[9] = x - y; + x = arr[2]; y = arr[10]; arr[2] = x + y; arr[10] = x - y; + x = arr[3]; y = arr[11]; arr[3] = x + y; arr[11] = x - y; + x = arr[4]; y = arr[12]; arr[4] = x + y; arr[12] = x - y; + x = arr[5]; y = arr[13]; arr[5] = x + y; arr[13] = x - y; + x = arr[6]; y = arr[14]; arr[6] = x + y; arr[14] = x - y; + x = arr[7]; y = arr[15]; arr[7] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[4]; arr[0] = x + y; arr[4] = x - y; + x = arr[1]; y = arr[5]; arr[1] = x + y; arr[5] = x - y; + x = arr[2]; y = arr[6]; arr[2] = x + y; arr[6] = x - y; + x = arr[3]; y = arr[7]; arr[3] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[12]; arr[8] = x + y; arr[12] = x - y; + x = arr[9]; y = arr[13]; arr[9] = x + y; arr[13] = x - y; + x = arr[10]; y = arr[14]; arr[10] = x + y; arr[14] = x - y; + x = arr[11]; y = arr[15]; arr[11] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[2]; arr[0] = x + y; arr[2] = x - y; + x = arr[1]; y = arr[3]; arr[1] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[6]; arr[4] = x + y; arr[6] = x - y; + x = arr[5]; y = arr[7]; arr[5] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[10]; arr[8] = x + y; arr[10] = x - y; + x = arr[9]; y = arr[11]; arr[9] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[14]; arr[12] = x + y; arr[14] = x - y; + x = arr[13]; y = arr[15]; arr[13] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[1]; arr[0] = x + y; arr[1] = x - y; + x = arr[2]; y = arr[3]; arr[2] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[5]; arr[4] = x + y; arr[5] = x - y; + x = arr[6]; y = arr[7]; arr[6] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[9]; arr[8] = x + y; arr[9] = x - y; + x = arr[10]; y = arr[11]; arr[10] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[13]; arr[12] = x + y; arr[13] = x - y; + x = arr[14]; y = arr[15]; arr[14] = x + y; arr[15] = x - y; + } + + /** + * Raw Fast Fourier Transform on 16 wide array of doubles + * @param arr is the 16-long array + */ + private static void hashFft16(double[] arr) { + double x,y; + + x = arr[0]; y = arr[8]; arr[0] = x + y; arr[8] = x - y; + x = arr[1]; y = arr[9]; arr[1] = x + y; arr[9] = x - y; + x = arr[2]; y = arr[10]; arr[2] = x + y; arr[10] = x - y; + x = arr[3]; y = arr[11]; arr[3] = x + y; arr[11] = x - y; + x = arr[4]; y = arr[12]; arr[4] = x + y; arr[12] = x - y; + x = arr[5]; y = arr[13]; arr[5] = x + y; arr[13] = x - y; + x = arr[6]; y = arr[14]; arr[6] = x + y; arr[14] = x - y; + x = arr[7]; y = arr[15]; arr[7] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[4]; arr[0] = x + y; arr[4] = x - y; + x = arr[1]; y = arr[5]; arr[1] = x + y; arr[5] = x - y; + x = arr[2]; y = arr[6]; arr[2] = x + y; arr[6] = x - y; + x = arr[3]; y = arr[7]; arr[3] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[12]; arr[8] = x + y; arr[12] = x - y; + x = arr[9]; y = arr[13]; arr[9] = x + y; arr[13] = x - y; + x = arr[10]; y = arr[14]; arr[10] = x + y; arr[14] = x - y; + x = arr[11]; y = arr[15]; arr[11] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[2]; arr[0] = x + y; arr[2] = x - y; + x = arr[1]; y = arr[3]; arr[1] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[6]; arr[4] = x + y; arr[6] = x - y; + x = arr[5]; y = arr[7]; arr[5] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[10]; arr[8] = x + y; arr[10] = x - y; + x = arr[9]; y = arr[11]; arr[9] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[14]; arr[12] = x + y; arr[14] = x - y; + x = arr[13]; y = arr[15]; arr[13] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[1]; arr[0] = x + y; arr[1] = x - y; + x = arr[2]; y = arr[3]; arr[2] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[5]; arr[4] = x + y; arr[5] = x - y; + x = arr[6]; y = arr[7]; arr[6] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[9]; arr[8] = x + y; arr[9] = x - y; + x = arr[10]; y = arr[11]; arr[10] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[13]; arr[12] = x + y; arr[13] = x - y; + x = arr[14]; y = arr[15]; arr[14] = x + y; arr[15] = x - y; + } + + public LSHBinner() { + doubleBuffer = new double[16]; + k = -1; + L = -1; + tokenList = null; + } + + public void setKandL(int k,int L) { + this.k = k; + this.L = L; + int numBits = 1; + while( (1 << numBits) <= L ) + numBits += 1; + numBits += k; + int numChar = numBits / 6; + if ((numBits % 6)!= 0) + numChar += 1; + tokenList = new BytesRef[L]; + for(int i=0;i>> 24) & 0x1f; + signPtr = rowNum * 16; + for (j = 0; j < 16; ++j) { // Based on the precalculated coeff table calculate this portion of dotproduct + if (hashSignTable[signPtr + j] == '+') + doubleBuffer[j] += vec[i].getCoeff(); // Dot product with +1 // coeff + else + doubleBuffer[j] -= vec[i].getCoeff(); // Dot product with -1 // coeff + } + } + } + else { // If we have many non-zero coefficients in -vec- + for (i = 0; i < vec.length; ++i) { + rowNum = vec[i].getHash() ^ hashcur; // Calculate the rest of the r_0 hashing function + rowNum = (rowNum * HASH_MULTIPLIER) + HASH_ADDEND; + rowNum = (rowNum >>> 24) & 0x1f; + if (rowNum < 0x10) // Set-up for the FFT + doubleBuffer[rowNum] += vec[i].getCoeff(); + else + doubleBuffer[rowNum & 0xf] -= vec[i].getCoeff(); + } + hashFft16(doubleBuffer); // Calculate the remaining dot-products be performing FFT + } + + for (i = 0; i < 16; ++i) { // Convert the dot-product results to a bit-vector + bucket <<= 1; + if (doubleBuffer[i] > 0.0) + bucket |= 1; + } + return bucket; + } + + public void generateBinIds(HashEntry[] vec) + + { + int bucket = 0; + int bucketcnt = 0; + int i,bitsleft; + int curid; + int mask,val; + int hashbase = LSH_HASHBASE; + + for (i = 0; i < L; ++i) { + curid = i; // Tack-on bits that indicate the particular table this bin id belongs to + bitsleft = k; + do { + if (bucketcnt == 0) { + hashbase = (hashbase * HASH_MULTIPLIER) + HASH_ADDEND; + bucket = hash16DotProduct(bucket, vec, hashbase); + bucketcnt += 16; + } + if (bucketcnt >= bitsleft) { + curid <<= bitsleft; + mask = 1; + mask = (mask << bitsleft) - 1; + val = bucket >>> (bucketcnt - bitsleft); + curid |= (val & mask); + bucketcnt -= bitsleft; + bitsleft = 0; + } else { + curid <<= bucketcnt; + mask = 1; + mask = (mask << bucketcnt) - 1; + curid |= (bucket & mask); + bitsleft -= bucketcnt; + bucketcnt = 0; + } + } while (bitsleft > 0); + char[] token = tokenList[i].buffer; + for(int j=0;j>= 6; // move to next 6 bits + } + } + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHTokenizer.java b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHTokenizer.java new file mode 100755 index 0000000000..281ddfdaeb --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHTokenizer.java @@ -0,0 +1,68 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugin.analysis.lsh; + +import java.io.IOException; + +import org.apache.lucene.analysis.Tokenizer; +import org.apache.lucene.analysis.tokenattributes.CharTermAttribute; +import org.elasticsearch.plugin.analysis.lsh.LSHBinner.BytesRef; + +import generic.lsh.vector.LSHVector; +import ghidra.features.bsim.query.elastic.Base64VectorFactory; + +public class LSHTokenizer extends Tokenizer { + private final CharTermAttribute bytesAtt = addAttribute(CharTermAttribute.class); + private BytesRef[] tokens; + private int pos; // Number of terms/tokens returned so far + private Base64VectorFactory vectorFactory; + private LSHBinner binner; + private char[] vecBuffer; + + public LSHTokenizer(int k,int L,Base64VectorFactory vFactory) { + super(DEFAULT_TOKEN_ATTRIBUTE_FACTORY); + vectorFactory = vFactory; + binner = new LSHBinner(); + binner.setKandL(k, L); + pos = -1; + vecBuffer = Base64VectorFactory.allocateBuffer(); + } + + @Override + public boolean incrementToken() throws IOException { + clearAttributes(); + if (pos < 0) { + LSHVector vector = vectorFactory.restoreVectorFromBase64(input,vecBuffer); +// AnalysisLSHPlugin.settingString = AnalysisLSHPlugin.settingString + " : " + Long.toHexString(vector.calcUniqueHash()); + binner.generateBinIds(vector.getEntries()); + tokens = binner.getTokenList(); + pos = 0; + } + if (pos < tokens.length) { + char[] buffer = tokens[pos].buffer; + bytesAtt.copyBuffer(buffer,0,buffer.length); + pos += 1; + return true; + } + return false; + } + + @Override + public void reset() throws IOException { + super.reset(); + pos = -1; + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHTokenizerFactory.java b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHTokenizerFactory.java new file mode 100755 index 0000000000..1918bfad49 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/LSHTokenizerFactory.java @@ -0,0 +1,44 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugin.analysis.lsh; + +import org.apache.lucene.analysis.Tokenizer; +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.env.Environment; +import org.elasticsearch.index.IndexSettings; +import org.elasticsearch.index.analysis.AbstractTokenizerFactory; + +import ghidra.features.bsim.query.elastic.Base64VectorFactory; +import ghidra.features.bsim.query.elastic.ElasticUtilities; + +public class LSHTokenizerFactory extends AbstractTokenizerFactory { + + private Base64VectorFactory vectorFactory; + private int k; + private int L; + + public LSHTokenizerFactory(IndexSettings indexSettings, Environment environment, String name, Settings settings) { + super(indexSettings, settings, name); + k = settings.getAsInt(ElasticUtilities.K_SETTING, -1); + L = settings.getAsInt(ElasticUtilities.L_SETTING, -1); + vectorFactory = AnalysisLSHPlugin.getVectorFactory(name); + } + + @Override + public Tokenizer create() { + return new LSHTokenizer(k,L,vectorFactory); + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java new file mode 100755 index 0000000000..e47b274018 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/src/org/elasticsearch/plugin/analysis/lsh/VectorCompareScriptFactory.java @@ -0,0 +1,147 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugin.analysis.lsh; + +import java.io.*; +import java.util.Map; + +import org.apache.lucene.document.Document; +import org.apache.lucene.util.BytesRef; +import org.elasticsearch.script.*; +import org.elasticsearch.script.ScoreScript.LeafFactory; +import org.elasticsearch.search.lookup.SearchLookup; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.VectorCompare; +import ghidra.features.bsim.query.elastic.Base64VectorFactory; + +public class VectorCompareScriptFactory implements ScoreScript.Factory { + + public final static String SCRIPT_NAME = "lsh_compare"; + public final static String FEATURES_NAME = "{\"features\":\""; + + @Override + public boolean isResultDeterministic() { + return true; + } + + @Override + public LeafFactory newFactory(Map params, SearchLookup lookup) { + return new VectorCompareLeafFactory(params, lookup); + } + + private static class VectorCompareLeafFactory implements LeafFactory { + + private final Map params; + private final SearchLookup lookup; + private LSHVector baseVector; // Vector being compared to everything + private final double simthresh; // Similarity threshold + private final double sigthresh; // Significance threshold + private final Base64VectorFactory vectorFactory; // Factory used for this particular query + + private VectorCompareLeafFactory(Map params, SearchLookup lookup) { + this.params = params; + this.lookup = lookup; + vectorFactory = AnalysisLSHPlugin.getVectorFactory((String) params.get("indexname")); + simthresh = (Double) params.get("simthresh"); + sigthresh = (Double) params.get("sigthresh"); + StringReader reader = new StringReader((String) params.get("vector")); + try { + baseVector = vectorFactory.restoreVectorFromBase64(reader, + Base64VectorFactory.allocateBuffer()); + } + catch (IOException e) { + baseVector = null; + } + } + + @Override + public boolean needs_score() { + return false; + } + + private static int scanForFeatures(byte[] buffer, int offset) throws IOException { + int i = 0; + while (i < FEATURES_NAME.length()) { + char curChar = FEATURES_NAME.charAt(i); + int val = buffer[offset]; + if (val == curChar) { + i += 1; + offset += 1; + } + else if (val == ' ' || val == '\t') { + offset += 1; + } + else { + throw new IOException("Document is missing \"features\""); + } + } + return offset; + } + + private static int scanForLength(BytesRef byteRef, int startOffset) throws IOException { + int finalLength = 0; + int maxLength = byteRef.length - (startOffset - byteRef.offset); + while (finalLength < maxLength) { + if (byteRef.bytes[finalLength + startOffset] == '\"') { + break; + } + finalLength += 1; + } + if (finalLength == byteRef.length) { + throw new IOException("Document does not contain complete \"features\""); + } + return finalLength; + } + + @Override + public ScoreScript newInstance(DocReader docReader) throws IOException { + return new ScoreScript(params, lookup, docReader) { + @Override + public double execute(ExplanationHolder explanation) { + try { + DocValuesDocReader dvReader = (DocValuesDocReader) docReader; + Document document = + dvReader.getLeafReaderContext().reader().document(_getDocId()); + BytesRef byteRef = document.getField("_source").binaryValue(); + int valOffset = scanForFeatures(byteRef.bytes, byteRef.offset); + int finalLength = scanForLength(byteRef, valOffset); + InputStream inputStream = + new ByteArrayInputStream(byteRef.bytes, valOffset, finalLength); + Reader reader = new InputStreamReader(inputStream); + // Should be sharing the VectorCompare between different calls + // but apparently this routine needs to be thread safe, so we allocate it per call + VectorCompare vectorCompare = new VectorCompare(); + LSHVector curVec = vectorFactory.restoreVectorFromBase64(reader, + Base64VectorFactory.allocateBuffer()); + double sim = baseVector.compare(curVec, vectorCompare); + if (sim <= simthresh) { + return 0.0; + } + double sig = vectorFactory.calculateSignificance(vectorCompare); + if (sig <= sigthresh) { + return 0.0; + } + return sim; + } + catch (IOException e) { + return 0.0; + } + } + }; + } + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/TokenStream.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/TokenStream.java new file mode 100644 index 0000000000..48b79c7597 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/TokenStream.java @@ -0,0 +1,29 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.analysis; + +import java.io.Closeable; +import java.io.IOException; + +import org.apache.lucene.util.AttributeFactory; +import org.apache.lucene.util.AttributeSource; + +public abstract class TokenStream extends AttributeSource implements Closeable { + public static final AttributeFactory DEFAULT_TOKEN_ATTRIBUTE_FACTORY = null; + + public abstract boolean incrementToken() throws IOException; +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/Tokenizer.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/Tokenizer.java new file mode 100644 index 0000000000..6328b71733 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/Tokenizer.java @@ -0,0 +1,38 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.analysis; + +import java.io.IOException; +import java.io.Reader; + +import org.apache.lucene.util.AttributeFactory; + +public abstract class Tokenizer extends TokenStream { + protected Reader input; + + protected Tokenizer(AttributeFactory factory) { + + } + + @Override + public void close() throws IOException { + } + + public void reset() throws IOException { + } + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/tokenattributes/CharTermAttribute.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/tokenattributes/CharTermAttribute.java new file mode 100644 index 0000000000..065762b4a3 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/analysis/tokenattributes/CharTermAttribute.java @@ -0,0 +1,25 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.analysis.tokenattributes; + +import org.apache.lucene.util.Attribute; + +public interface CharTermAttribute extends Attribute, CharSequence, Appendable { + + public void copyBuffer(char[] buffer, int offset, int length); + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/document/Document.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/document/Document.java new file mode 100644 index 0000000000..64283f632f --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/document/Document.java @@ -0,0 +1,26 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.document; + +import org.apache.lucene.index.IndexableField; + +public class Document { + public final IndexableField getField(String name) { + return null; + } + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexReader.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexReader.java new file mode 100644 index 0000000000..5d6f5e0705 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexReader.java @@ -0,0 +1,27 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import java.io.Closeable; +import java.io.IOException; + +import org.apache.lucene.document.Document; + +public abstract class IndexReader implements Closeable { + public final Document document(int docID) throws IOException { + return null; + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexReaderContext.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexReaderContext.java new file mode 100644 index 0000000000..3a6103f77c --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexReaderContext.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +public abstract class IndexReaderContext { + public abstract IndexReader reader(); + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexableField.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexableField.java new file mode 100644 index 0000000000..6e497d803f --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/IndexableField.java @@ -0,0 +1,23 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +import org.apache.lucene.util.BytesRef; + +public interface IndexableField { + public BytesRef binaryValue(); +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/LeafReader.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/LeafReader.java new file mode 100644 index 0000000000..a11eff15d9 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/LeafReader.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +public abstract class LeafReader extends IndexReader { + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/LeafReaderContext.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/LeafReaderContext.java new file mode 100644 index 0000000000..2c5d17c5f8 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/index/LeafReaderContext.java @@ -0,0 +1,24 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.index; + +public final class LeafReaderContext extends IndexReaderContext { + @Override + public LeafReader reader() { + return null; + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/Attribute.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/Attribute.java new file mode 100644 index 0000000000..343bf8c0a0 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/Attribute.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.util; + +public interface Attribute { + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/AttributeFactory.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/AttributeFactory.java new file mode 100644 index 0000000000..8e0e605de2 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/AttributeFactory.java @@ -0,0 +1,20 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.util; + +public abstract class AttributeFactory { + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/AttributeSource.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/AttributeSource.java new file mode 100644 index 0000000000..9087926431 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/AttributeSource.java @@ -0,0 +1,27 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.util; + +public class AttributeSource { + public final T addAttribute(Class attClass) { + return null; + } + + public final void clearAttributes() { + + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/BytesRef.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/BytesRef.java new file mode 100644 index 0000000000..a3d7b82aaf --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/apache/lucene/util/BytesRef.java @@ -0,0 +1,23 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for lucene class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.apache.lucene.util; + +public class BytesRef { + public byte[] bytes; + public int length; + public int offset; +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/common/settings/Settings.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/common/settings/Settings.java new file mode 100644 index 0000000000..3cd9422cb2 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/common/settings/Settings.java @@ -0,0 +1,34 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.common.settings; + +import java.util.Set; + +public class Settings { + + public Integer getAsInt(String setting, Integer defaultValue) { + return null; + } + + public String get(String setting) { + return null; + } + + public Set keySet() { + return null; + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/env/Environment.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/env/Environment.java new file mode 100644 index 0000000000..6224f60350 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/env/Environment.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.env; + +public class Environment { + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/IndexModule.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/IndexModule.java new file mode 100644 index 0000000000..07115ce131 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/IndexModule.java @@ -0,0 +1,26 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.index; + +import org.elasticsearch.common.settings.Settings; + +public class IndexModule { + + public Settings getSettings() { + return null; + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/IndexSettings.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/IndexSettings.java new file mode 100644 index 0000000000..a6b9d797b1 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/IndexSettings.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.index; + +public final class IndexSettings { + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/analysis/AbstractTokenizerFactory.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/analysis/AbstractTokenizerFactory.java new file mode 100644 index 0000000000..86f61013d5 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/analysis/AbstractTokenizerFactory.java @@ -0,0 +1,27 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.index.analysis; + +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.index.IndexSettings; + +public abstract class AbstractTokenizerFactory implements TokenizerFactory { + + public AbstractTokenizerFactory(IndexSettings indexSettings, Settings settings, String name) { + + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/analysis/TokenizerFactory.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/analysis/TokenizerFactory.java new file mode 100644 index 0000000000..6fef2040a5 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/index/analysis/TokenizerFactory.java @@ -0,0 +1,24 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.index.analysis; + +import org.apache.lucene.analysis.Tokenizer; + +public interface TokenizerFactory { + Tokenizer create(); + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/indices/analysis/AnalysisModule.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/indices/analysis/AnalysisModule.java new file mode 100644 index 0000000000..e121bfc67c --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/indices/analysis/AnalysisModule.java @@ -0,0 +1,31 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.indices.analysis; + +import java.io.IOException; + +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.env.Environment; +import org.elasticsearch.index.IndexSettings; + +public class AnalysisModule { + + public interface AnalysisProvider { + T get(IndexSettings indexSettings, Environment environment, String name, Settings settings) + throws IOException; + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/AnalysisPlugin.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/AnalysisPlugin.java new file mode 100644 index 0000000000..b3a8e0be89 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/AnalysisPlugin.java @@ -0,0 +1,27 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugins; + +import java.util.Map; + +import org.elasticsearch.index.analysis.TokenizerFactory; +import org.elasticsearch.indices.analysis.AnalysisModule.AnalysisProvider; + +public interface AnalysisPlugin { + Map> getTokenizers(); + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/Plugin.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/Plugin.java new file mode 100644 index 0000000000..ee934af4c9 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/Plugin.java @@ -0,0 +1,32 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugins; + +import java.io.Closeable; +import java.io.IOException; + +import org.elasticsearch.index.IndexModule; + +public abstract class Plugin implements Closeable { + public void onIndexModule(IndexModule indexModule) { + } + + @Override + public void close() throws IOException { + + } +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/ScriptPlugin.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/ScriptPlugin.java new file mode 100644 index 0000000000..4354ee7fb4 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/plugins/ScriptPlugin.java @@ -0,0 +1,28 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.plugins; + +import java.util.Collection; + +import org.elasticsearch.common.settings.Settings; +import org.elasticsearch.script.ScriptContext; +import org.elasticsearch.script.ScriptEngine; + +public interface ScriptPlugin { + ScriptEngine getScriptEngine(Settings settings, Collection> contexts); + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/DocReader.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/DocReader.java new file mode 100644 index 0000000000..cbfccbb690 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/DocReader.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.script; + +public interface DocReader { + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/DocValuesDocReader.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/DocValuesDocReader.java new file mode 100644 index 0000000000..d13a893ea1 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/DocValuesDocReader.java @@ -0,0 +1,28 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.script; + +import org.apache.lucene.index.LeafReaderContext; + +public class DocValuesDocReader implements DocReader, LeafReaderContextSupplier { + + @Override + public LeafReaderContext getLeafReaderContext() { + return null; + } + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/LeafReaderContextSupplier.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/LeafReaderContextSupplier.java new file mode 100644 index 0000000000..e9f391c07a --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/LeafReaderContextSupplier.java @@ -0,0 +1,23 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.script; + +import org.apache.lucene.index.LeafReaderContext; + +public interface LeafReaderContextSupplier { + LeafReaderContext getLeafReaderContext(); +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScoreScript.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScoreScript.java new file mode 100644 index 0000000000..01f5111046 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScoreScript.java @@ -0,0 +1,50 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.script; + +import java.io.IOException; +import java.util.Map; + +import org.elasticsearch.search.lookup.SearchLookup; + +public abstract class ScoreScript { + public ScoreScript(Map params, SearchLookup searchLookup, DocReader docReader) { + + } + + public static class ExplanationHolder { + + } + + public static final ScriptContext CONTEXT = null; + + public interface Factory extends ScriptFactory { + LeafFactory newFactory(Map params, SearchLookup lookup); + } + + public interface LeafFactory { + boolean needs_score(); + + ScoreScript newInstance(DocReader reader) throws IOException; + } + + public int _getDocId() { + return 0; + } + + public abstract double execute(ExplanationHolder explanation); +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptContext.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptContext.java new file mode 100644 index 0000000000..ff787a58bb --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptContext.java @@ -0,0 +1,22 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.script; + +public final class ScriptContext { + public final String name = null; + public final Class factoryClazz = null; +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptEngine.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptEngine.java new file mode 100644 index 0000000000..3ebb55d6c1 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptEngine.java @@ -0,0 +1,30 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch interface + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.script; + +import java.io.Closeable; +import java.util.Map; +import java.util.Set; + +public interface ScriptEngine extends Closeable { + String getType(); + + FactoryType compile(String name, String code, ScriptContext context, + Map params); + + Set> getSupportedContexts(); +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptFactory.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptFactory.java new file mode 100644 index 0000000000..48ffad0183 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/script/ScriptFactory.java @@ -0,0 +1,22 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.script; + +public interface ScriptFactory { + boolean isResultDeterministic(); + +} diff --git a/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/search/lookup/SearchLookup.java b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/search/lookup/SearchLookup.java new file mode 100644 index 0000000000..5135888ff0 --- /dev/null +++ b/Ghidra/Extensions/BSimElasticPlugin/srcdummy/org/elasticsearch/search/lookup/SearchLookup.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * NOTE: Dummy placeholder for elasticsearch class + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package org.elasticsearch.search.lookup; + +public class SearchLookup { + +} diff --git a/Ghidra/Features/BSim/Module.manifest b/Ghidra/Features/BSim/Module.manifest new file mode 100755 index 0000000000..9df5107b96 --- /dev/null +++ b/Ghidra/Features/BSim/Module.manifest @@ -0,0 +1,9 @@ +##MODULE IP: Oxygen Icons - LGPL 3.0 +MODULE FILE LICENSE: postgresql-15.3.tar.gz Postgresql License +MODULE FILE LICENSE: lib/postgresql-42.6.0.jar PostgresqlJDBC License +MODULE FILE LICENSE: lib/json-simple-1.1.1.jar Apache License 2.0 +MODULE FILE LICENSE: lib/commons-dbcp2-2.9.0.jar Apache License 2.0 +MODULE FILE LICENSE: lib/commons-pool2-2.11.1.jar Apache License 2.0 +MODULE FILE LICENSE: lib/commons-logging-1.2.jar Apache License 2.0 +MODULE FILE LICENSE: lib/log4j-jcl-2.16.0.jar Apache License 2.0 +MODULE FILE LICENSE: lib/h2-2.2.220.jar H2 Mozilla License 2.0 \ No newline at end of file diff --git a/Ghidra/Features/BSim/build.gradle b/Ghidra/Features/BSim/build.gradle new file mode 100755 index 0000000000..b1b9ba0fd7 --- /dev/null +++ b/Ghidra/Features/BSim/build.gradle @@ -0,0 +1,197 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +apply from: "$rootProject.projectDir/gradle/distributableGhidraModule.gradle" +apply from: "$rootProject.projectDir/gradle/javaProject.gradle" +apply from: "$rootProject.projectDir/gradle/javaTestProject.gradle" +apply from: "$rootProject.projectDir/gradle/nativeProject.gradle" +apply from: "$rootProject.projectDir/gradle/helpProject.gradle" + +apply plugin: 'eclipse' +eclipse.project.name = 'Features BSim' + +import java.nio.file.Files +import org.gradle.util.GUtil + +// NOTE: fetchDependencies.gradle must be updated if postgresql version changes +def postgresql_distro = "postgresql-15.3.tar.gz" + +dependencies { + api project(":Decompiler") + api project(":CodeCompare") + + api "org.postgresql:postgresql:42.6.0" + api "org.json.simple:json-simple:1.1.1" + api "org.apache.commons:commons-dbcp2:2.9.0" + api "org.apache.commons:commons-pool2:2.11.1" + api "org.apache.commons:commons-logging:1.2" + api "org.apache.logging.log4j:log4j-jcl:2.16.0" + api "com.h2database:h2:2.2.220" +} + +// Copy postgresql source distro, lshvector plugin source, and make-postgres.sh +// into common zip to allow for a rebuild of the postgres server if needed + +rootProject.assembleDistribution { + + String postgresqlDepsFile = "${DEPS_DIR}/BSim/${postgresql_distro}" + String postgresqlBinRepoFile = "${BIN_REPO}/Ghidra/Features/BSim/${postgresql_distro}" + + def postgresqlFile = file(postgresqlDepsFile).exists() ? postgresqlDepsFile : postgresqlBinRepoFile + + into (getZipPath(this.project)) { + from file("make-postgres.sh") + } + into (getZipPath(this.project)) { + from file(postgresqlFile) + } + into (getZipPath(this.project) + "/src/lshvector") { + from files("src/lshvector") + } +} + +// Relative to the 'workingDir' Exec task property. +def installPoint = "../help/help" + +/** + * Build the pdf docs for BSim and place into the '$installPoint' directory. + * A build (ex: 'gradle buildLocalTSSI_Release') will place the pdf in the distribution. + * There is an associated, auto-generated clean task. + **/ +task buildBSimHelpPdf(type: Exec) { + + workingDir 'src/main/doc' + + def buildDir = "../../../build/BSimDocumentationPdf" + + // Gradle will provide a cleanBuildBSimDocumentationPdf task that will remove these + // declared outputs. + outputs.dir "$workingDir/$buildDir" + outputs.file "$workingDir/$buildDir/bsim.pdf" + + // 'which' returns the number of failed arguments + // Using the 'which' command first will allow the task to fail if the required + // executables are not installed. + // + // The bash commands end with "2>&1" to redirect stderr to stdout and have all + // messages print in sequence + // + // 'commandLine' takes one command, so wrap multiple commands in bash. + commandLine 'bash', '-e', '-c', """ + echo '** Checking if required executables are installed. **' + which xsltproc + which fop + + echo '** Preparing for xsltproc **' + mkdir -p $buildDir/images + + cp $installPoint/topics/BSimDatabasePlugin/images/*.png $buildDir/images + + echo '** Building bsim.fo **' + xsltproc --output $buildDir/bsim_withscaling.xml --stringparam profile.condition "withscaling" commonprofile.xsl bsim.xml 2>&1 + xsltproc --output $buildDir/bsim.fo focustom.xsl $buildDir/bsim_withscaling.xml 2>&1 + + echo '** Building bsim.pdf **' + fop $buildDir/bsim.fo $buildDir/bsim.pdf 2>&1 + + echo '** Done. **' + """ + + // Allows doLast block regardless of exit value. + ignoreExitValue true + + // Store the output instead of printing to the console. + standardOutput = new ByteArrayOutputStream() + ext.output = { standardOutput.toString() } + ext.errorOutput = { standardOutput.toString() } + + // Check the OS before executing command. + doFirst { + if (!getCurrentPlatformName().startsWith("linux")) { + throw new TaskExecutionException( it, new Exception("The '$it.name' task only works on Linux.")) + } + } + + // Print the output of the commands and check the return value. + doLast { + println output() + if (execResult.exitValue) { + logger.error("$it.name: An error occurred. Here is the output:\n" + output()) + throw new TaskExecutionException( it, new Exception("'$it.name': The command: '${commandLine.join(' ')}'" + + " task \nfailed with exit code $execResult.exitValue; see task output for details.")) + } + } +} + +/** + * Build the html docs for BSim and place into the '$installPoint' directory. + * A build (ex: 'gradle buildLocalTSSI_Release') will place the html files in the distribution. + **/ +task buildBSimHelpHtml(type: Exec) { + + workingDir 'src/main/doc' + + def buildDir = "../../../build/html" + + // 'which' returns the number of failed arguments + // Using the 'which' command first will allow the task to fail if the required + // executables are not installed. + // + // The bash commands end with "2>&1" to redirect stderr to stdout and have all + // messages print in sequence + // + // 'commandLine' takes one command, so wrap multiple commands in bash. + commandLine 'bash', '-e', '-c', """ + echo '** Checking if required executables are installed. **' + which xsltproc + which sed + + echo '** Removing older html files installed under '$installPoint' **' + rm -f $installPoint/topics/BSimDatabasePlugin/*.html + + echo '** Building html files **' + xsltproc --output $buildDir/bsim_noscaling.xml --stringparam profile.condition "noscaling" commonprofile.xsl bsim.xml 2>&1 + xsltproc --stringparam base.dir ${installPoint}/topics/BSimDatabasePlugin/ htmlcustom.xsl $buildDir/bsim_noscaling.xml 2>&1 + sed -i -e '/DefaultStyle.css/ { p; sQhref=".*"Qhref="../../shared/languages.css"Q; }' ${installPoint}/topics/BSimDatabasePlugin/*.html + rm $installPoint/topics/BSimDatabasePlugin/index.html + + echo '** Done. **' + """ + + // Allows doLast block regardless of exit value. + ignoreExitValue true + + // Store the output instead of printing to the console. + standardOutput = new ByteArrayOutputStream() + ext.output = { standardOutput.toString() } + ext.errorOutput = { standardOutput.toString() } + + // Check the OS before executing command. + doFirst { + if (!getCurrentPlatformName().startsWith("linux")) { + throw new TaskExecutionException( it, new Exception("The '$it.name' task only works on Linux.")) + } + } + + // Print the output of the commands and check the return value. + doLast { + println output() + if (execResult.exitValue) { + logger.error("$it.name: An error occurred. Here is the output:\n" + output()) + throw new TaskExecutionException( it, new Exception("'$it.name': The command: '${commandLine.join(' ')}'" + + " task \nfailed with exit code $execResult.exitValue; see task output for details.")) + } + } +} diff --git a/Ghidra/Features/BSim/certification.manifest b/Ghidra/Features/BSim/certification.manifest new file mode 100755 index 0000000000..12c5480cf9 --- /dev/null +++ b/Ghidra/Features/BSim/certification.manifest @@ -0,0 +1,51 @@ +##VERSION: 2.0 +##MODULE IP: Apache License 2.0 +##MODULE IP: Creative Commons Attribution 2.5 +##MODULE IP: Crystal Clear Icons - LGPL 2.1 +##MODULE IP: FAMFAMFAM Icons - CC 2.5 +##MODULE IP: H2 Mozilla License 2.0 +##MODULE IP: LGPL 2.1 +##MODULE IP: LGPL 3.0 +##MODULE IP: Oxygen Icons - LGPL 3.0 +##MODULE IP: Postgresql License +##MODULE IP: PostgresqlJDBC License +##MODULE IP: Public Domain +Module.manifest||GHIDRA||||END| +data/bsim.theme.properties||GHIDRA||||END| +data/large_32.xml||GHIDRA||||END| +data/lshweights_32.xml||GHIDRA|||Signature data|END| +data/lshweights_64.xml||GHIDRA|||Signature data|END| +data/lshweights_64_32.xml||GHIDRA|||Signature data|END| +data/lshweights_cpool.xml||GHIDRA||||END| +data/lshweights_nosize.xml||GHIDRA||||END| +data/medium_32.xml||GHIDRA||||END| +data/medium_64.xml||GHIDRA||||END| +data/medium_cpool.xml||GHIDRA||||END| +data/medium_nosize.xml||GHIDRA||||END| +data/serverconfig.xml||GHIDRA||||END| +src/lshvector/Makefile.lshvector||GHIDRA||||END| +src/lshvector/lshvector--1.0.sql||GHIDRA||||END| +src/lshvector/lshvector.control||GHIDRA||||END| +src/main/help/help/TOC_Source.xml||GHIDRA||||END| +src/main/help/help/topics/BSim/BSimOverview.html||GHIDRA||||END| +src/main/help/help/topics/BSim/CommandLineReference.html||GHIDRA||||END| +src/main/help/help/topics/BSim/DatabaseConfiguration.html||GHIDRA||||END| +src/main/help/help/topics/BSim/FeatureWeight.html||GHIDRA||||END| +src/main/help/help/topics/BSim/IngestProcess.html||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/BSimSearch.html||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/images/AddServerDialog.png||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/images/ApplyResultsPanel.png||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewDialog.png||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewResults.png||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/images/BSimResultsProvider.png||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/images/BSimSearchDialog.png||GHIDRA||||END| +src/main/help/help/topics/BSimSearchPlugin/images/ManageServersDialog.png||GHIDRA||||END| +src/main/resources/bsim.log4j.xml||GHIDRA||||END| +src/main/resources/images/checkmark_yellow.gif||GHIDRA||||END| +src/main/resources/images/flag_green.png||FAMFAMFAM Icons - CC 2.5|||famfamfam silk icon set|END| +src/main/resources/images/preferences-desktop-user-password.png||Oxygen Icons - LGPL 3.0|||Oxygen icon theme (dual license; LGPL or CC-SA-3.0)|END| +src/main/resources/images/preferences-web-browser-shortcuts-32.png||Oxygen Icons - LGPL 3.0|||Oxygen icon theme (dual license; LGPL or CC-SA-3.0)|END| +src/main/resources/images/preferences-web-browser-shortcuts.png||LGPL 3.0|||oxygen|END| +src/main/resources/images/view_top_bottom.png||Crystal Clear Icons - LGPL 2.1||||END| +src/main/resources/log4j-appender-console.xml||GHIDRA||||END| +src/main/resources/log4j-appender-rolling-file.xml||GHIDRA||||END| diff --git a/Ghidra/Features/BSim/data/bsim.theme.properties b/Ghidra/Features/BSim/data/bsim.theme.properties new file mode 100644 index 0000000000..c0efedbea0 --- /dev/null +++ b/Ghidra/Features/BSim/data/bsim.theme.properties @@ -0,0 +1,17 @@ + +[Defaults] + +icon.bsim.query.dialog.provider = preferences-web-browser-shortcuts.png + +icon.bsim.change.password = preferences-desktop-user-password.png + +icon.bsim.table.split = view_top_bottom.png + +icon.bsim.results.status.name.applied = checkmark_green.gif +icon.bsim.results.status.signature.applied = EMPTY_ICON {checkmark_green.gif[move(-2,-1)]} {checkmark_green.gif [move(4,0)]} +icon.bsim.results.status.matches = flag_green.png +icon.bsim.results.status.ignored = checkmark_yellow.gif + +icon.bsim.functions.table = FunctionScope.gif + +[Dark Defaults] diff --git a/Ghidra/Features/BSim/data/large_32.xml b/Ghidra/Features/BSim/data/large_32.xml new file mode 100755 index 0000000000..c1f2262fc1 --- /dev/null +++ b/Ghidra/Features/BSim/data/large_32.xml @@ -0,0 +1,13 @@ + + + Large 32-bit + Example Owner + A large (~100 million functions) database tuned for 32-bit executables + 0 + 0 + 0x49 + +19 +232 +lshweights_32.xml + diff --git a/Ghidra/Features/BSim/data/lshweights_32.xml b/Ghidra/Features/BSim/data/lshweights_32.xml new file mode 100755 index 0000000000..b2538a1cfd --- /dev/null +++ b/Ghidra/Features/BSim/data/lshweights_32.xml @@ -0,0 +1,1587 @@ + + + 1.00000000e+00 + 9.99461385e-01 + 9.98922480e-01 + 9.98383284e-01 + 9.97843797e-01 + 9.97304018e-01 + 9.96763946e-01 + 9.96223582e-01 + 9.95682925e-01 + 9.95141974e-01 + 9.94600728e-01 + 9.94059188e-01 + 9.93517353e-01 + 9.92975222e-01 + 9.92432795e-01 + 9.91890071e-01 + 9.91347050e-01 + 9.90803731e-01 + 9.90260115e-01 + 9.89716200e-01 + 9.89171985e-01 + 9.88627472e-01 + 9.88082658e-01 + 9.87537543e-01 + 9.86992128e-01 + 9.86446410e-01 + 9.85900391e-01 + 9.85354070e-01 + 9.84807445e-01 + 9.84260516e-01 + 9.83713284e-01 + 9.83165747e-01 + 9.82617905e-01 + 9.82069757e-01 + 9.81521303e-01 + 9.80972542e-01 + 9.80423475e-01 + 9.79874099e-01 + 9.79324416e-01 + 9.78774424e-01 + 9.78224122e-01 + 9.77673511e-01 + 9.77122589e-01 + 9.76571357e-01 + 9.76019814e-01 + 9.75467958e-01 + 9.74915790e-01 + 9.74363310e-01 + 9.73810515e-01 + 9.73257407e-01 + 9.72703985e-01 + 9.72150247e-01 + 9.71596193e-01 + 9.71041824e-01 + 9.70487138e-01 + 9.69932135e-01 + 9.69376813e-01 + 9.68821174e-01 + 9.68265216e-01 + 9.67708938e-01 + 9.67152341e-01 + 9.66595422e-01 + 9.66038183e-01 + 9.65480622e-01 + 9.64922739e-01 + 9.64364534e-01 + 9.63806005e-01 + 9.63617038e-01 + 9.63527373e-01 + 9.63170062e-01 + 9.62992210e-01 + 9.62903483e-01 + 9.62814889e-01 + 9.62549896e-01 + 9.62286079e-01 + 9.62110850e-01 + 9.61848970e-01 + 9.61761933e-01 + 9.61675022e-01 + 9.61501582e-01 + 9.61415050e-01 + 9.61242365e-01 + 9.61070179e-01 + 9.60984272e-01 + 9.60898490e-01 + 9.60812830e-01 + 9.60727294e-01 + 9.60641881e-01 + 9.60386372e-01 + 9.60301446e-01 + 9.60131956e-01 + 9.59962946e-01 + 9.59878620e-01 + 9.59794414e-01 + 9.59710326e-01 + 9.59626357e-01 + 9.59458773e-01 + 9.59375157e-01 + 9.59291658e-01 + 9.59208276e-01 + 9.59125010e-01 + 9.58958826e-01 + 9.58627841e-01 + 9.58545381e-01 + 9.58134779e-01 + 9.58052996e-01 + 9.57971325e-01 + 9.57889766e-01 + 9.57808317e-01 + 9.57726980e-01 + 9.57645753e-01 + 9.57483630e-01 + 9.57321946e-01 + 9.57241267e-01 + 9.57080236e-01 + 9.56999882e-01 + 9.56919637e-01 + 9.56759468e-01 + 9.56440411e-01 + 9.56360911e-01 + 9.56281518e-01 + 9.56202229e-01 + 9.56043966e-01 + 9.55964992e-01 + 9.55807354e-01 + 9.55493317e-01 + 9.55336914e-01 + 9.55180919e-01 + 9.55103073e-01 + 9.54792696e-01 + 9.54715352e-01 + 9.54330122e-01 + 9.54176720e-01 + 9.54100166e-01 + 9.53794922e-01 + 9.53642880e-01 + 9.53567003e-01 + 9.53415536e-01 + 9.53189050e-01 + 9.53113744e-01 + 9.52888391e-01 + 9.52813460e-01 + 9.52738623e-01 + 9.52663879e-01 + 9.52589228e-01 + 9.52365829e-01 + 9.52217356e-01 + 9.52143256e-01 + 9.51995331e-01 + 9.51921505e-01 + 9.51407240e-01 + 9.51334131e-01 + 9.51042579e-01 + 9.50897331e-01 + 9.50824837e-01 + 9.50680113e-01 + 9.50535734e-01 + 9.50319814e-01 + 9.50248012e-01 + 9.49890278e-01 + 9.49818986e-01 + 9.49676653e-01 + 9.49392990e-01 + 9.49322282e-01 + 9.49181114e-01 + 9.49040276e-01 + 9.48969980e-01 + 9.48899766e-01 + 9.48759582e-01 + 9.48480185e-01 + 9.48202075e-01 + 9.48132747e-01 + 9.47649665e-01 + 9.47580968e-01 + 9.47306955e-01 + 9.47170412e-01 + 9.47034177e-01 + 9.46830397e-01 + 9.46762623e-01 + 9.46694925e-01 + 9.46424885e-01 + 9.46290316e-01 + 9.46089020e-01 + 9.45489115e-01 + 9.45356605e-01 + 9.45290459e-01 + 9.45158382e-01 + 9.44960804e-01 + 9.44502277e-01 + 9.44306820e-01 + 9.44241808e-01 + 9.44176865e-01 + 9.44111991e-01 + 9.43917785e-01 + 9.43724198e-01 + 9.43531226e-01 + 9.43467038e-01 + 9.43338864e-01 + 9.43274879e-01 + 9.42892374e-01 + 9.42765405e-01 + 9.42386081e-01 + 9.42323090e-01 + 9.42197302e-01 + 9.41200248e-01 + 9.41138471e-01 + 9.41076756e-01 + 9.40891986e-01 + 9.40646494e-01 + 9.40463021e-01 + 9.40097722e-01 + 9.39915890e-01 + 9.39855399e-01 + 9.39674286e-01 + 9.39614034e-01 + 9.39433635e-01 + 9.38955158e-01 + 9.38598743e-01 + 9.38480399e-01 + 9.37950665e-01 + 9.37775101e-01 + 9.37716692e-01 + 9.37541797e-01 + 9.37135643e-01 + 9.36904759e-01 + 9.36732165e-01 + 9.36674741e-01 + 9.36560056e-01 + 9.36331326e-01 + 9.36217280e-01 + 9.35989824e-01 + 9.35819783e-01 + 9.35593794e-01 + 9.35312472e-01 + 9.35256362e-01 + 9.35144296e-01 + 9.34698065e-01 + 9.34642514e-01 + 9.34587013e-01 + 9.34255054e-01 + 9.33924878e-01 + 9.33815213e-01 + 9.33541901e-01 + 9.33215523e-01 + 9.32890867e-01 + 9.32783027e-01 + 9.32729178e-01 + 9.32675377e-01 + 9.32621622e-01 + 9.32567914e-01 + 9.32407070e-01 + 9.32353549e-01 + 9.32300074e-01 + 9.31873938e-01 + 9.31556259e-01 + 9.31503471e-01 + 9.31450729e-01 + 9.31345379e-01 + 9.31292771e-01 + 9.31030404e-01 + 9.30978064e-01 + 9.30612923e-01 + 9.30508993e-01 + 9.30457093e-01 + 9.30353425e-01 + 9.30198248e-01 + 9.30043459e-01 + 9.29991949e-01 + 9.29940482e-01 + 9.29735042e-01 + 9.29683788e-01 + 9.29632577e-01 + 9.29122786e-01 + 9.29021330e-01 + 9.28970664e-01 + 9.28869458e-01 + 9.28818916e-01 + 9.28416065e-01 + 9.28215619e-01 + 9.27469671e-01 + 9.27321553e-01 + 9.27124612e-01 + 9.26391605e-01 + 9.26246037e-01 + 9.25907701e-01 + 9.25715190e-01 + 9.25236510e-01 + 9.23775602e-01 + 9.23682517e-01 + 9.23589572e-01 + 9.21849456e-01 + 9.21804313e-01 + 9.21714125e-01 + 9.21354670e-01 + 9.21309883e-01 + 9.21175715e-01 + 9.20908241e-01 + 9.20464981e-01 + 9.20332614e-01 + 9.20200527e-01 + 9.19674949e-01 + 9.19500734e-01 + 9.19153751e-01 + 9.19067305e-01 + 9.18937859e-01 + 9.18894770e-01 + 9.18808681e-01 + 9.18465501e-01 + 9.18166754e-01 + 9.17278959e-01 + 9.16652425e-01 + 9.16486400e-01 + 9.16196908e-01 + 9.15908743e-01 + 9.15703717e-01 + 9.15255000e-01 + 9.15214366e-01 + 9.15173758e-01 + 9.14890234e-01 + 9.13570554e-01 + 9.13491455e-01 + 9.12744922e-01 + 9.12588876e-01 + 9.12200442e-01 + 9.12123041e-01 + 9.11354204e-01 + 9.11125365e-01 + 9.10935297e-01 + 9.10821529e-01 + 9.10745798e-01 + 9.10632371e-01 + 9.09434960e-01 + 9.09028523e-01 + 9.08624675e-01 + 9.08478457e-01 + 9.08187027e-01 + 9.08041813e-01 + 9.08005561e-01 + 9.07572147e-01 + 9.07428330e-01 + 9.06998824e-01 + 9.06785155e-01 + 9.06749613e-01 + 9.05937585e-01 + 9.05727445e-01 + 9.05309235e-01 + 9.04859252e-01 + 9.04378181e-01 + 9.04139001e-01 + 9.04104905e-01 + 9.03866745e-01 + 9.03460527e-01 + 9.03426792e-01 + 9.03157551e-01 + 9.02889440e-01 + 9.02789188e-01 + 9.02755806e-01 + 9.01893926e-01 + 9.01762355e-01 + 9.00590223e-01 + 9.00204231e-01 + 8.99948189e-01 + 8.99439151e-01 + 8.99123040e-01 + 8.98714410e-01 + 8.97258367e-01 + 8.96983272e-01 + 8.96618299e-01 + 8.96225231e-01 + 8.95744723e-01 + 8.95446215e-01 + 8.95416440e-01 + 8.95238079e-01 + 8.94912358e-01 + 8.94470825e-01 + 8.94441497e-01 + 8.93596714e-01 + 8.92392478e-01 + 8.92109028e-01 + 8.92024235e-01 + 8.91489770e-01 + 8.90710083e-01 + 8.89585039e-01 + 8.88881561e-01 + 8.87866895e-01 + 8.87444353e-01 + 8.86998385e-01 + 8.86529485e-01 + 8.86063931e-01 + 8.86038164e-01 + 8.85601674e-01 + 8.85550514e-01 + 8.85117260e-01 + 8.85091863e-01 + 8.84914365e-01 + 8.84159069e-01 + 8.82551890e-01 + 8.81582091e-01 + 8.80484248e-01 + 8.79918657e-01 + 8.79895196e-01 + 8.79824863e-01 + 8.78732675e-01 + 8.78067693e-01 + 8.77953716e-01 + 8.76914259e-01 + 8.76200688e-01 + 8.76067746e-01 + 8.75670518e-01 + 8.75560599e-01 + 8.75494735e-01 + 8.75472795e-01 + 8.74970169e-01 + 8.74666080e-01 + 8.74126512e-01 + 8.74062058e-01 + 8.73081524e-01 + 8.71203971e-01 + 8.70814285e-01 + 8.69839850e-01 + 8.69759287e-01 + 8.69618535e-01 + 8.68978851e-01 + 8.68542598e-01 + 8.67678546e-01 + 8.67659037e-01 + 8.65907170e-01 + 8.65831206e-01 + 8.65226575e-01 + 8.65132592e-01 + 8.65001236e-01 + 8.64496973e-01 + 8.63885749e-01 + 8.63701622e-01 + 8.62861072e-01 + 8.62553072e-01 + 8.61815994e-01 + 8.60786874e-01 + 8.58586778e-01 + 8.57571734e-01 + 8.57420781e-01 + 8.56737325e-01 + 8.56637882e-01 + 8.54006506e-01 + 8.53309039e-01 + 8.51549829e-01 + 8.50391620e-01 + 8.49119055e-01 + 8.48132362e-01 + 8.48044690e-01 + 8.47724179e-01 + 8.46599717e-01 + 8.45860159e-01 + 8.45803598e-01 + 8.45634192e-01 + 8.44487596e-01 + 8.43033093e-01 + 8.41875105e-01 + 8.41170988e-01 + 8.39861629e-01 + 8.39512297e-01 + 8.35597858e-01 + 8.34328390e-01 + 8.33743763e-01 + 8.32834811e-01 + 8.31683204e-01 + 8.30165224e-01 + 8.30108792e-01 + 8.26042637e-01 + 8.25322071e-01 + 8.25164105e-01 + 8.24410725e-01 + 8.24348303e-01 + 8.24161366e-01 + 8.24005960e-01 + 8.21404770e-01 + 8.20031211e-01 + 8.19426965e-01 + 8.15960307e-01 + 8.15637408e-01 + 8.13479317e-01 + 8.11880280e-01 + 8.10994519e-01 + 8.09052921e-01 + 8.08059820e-01 + 8.06802599e-01 + 8.06000525e-01 + 8.03404456e-01 + 8.01561629e-01 + 7.95397084e-01 + 7.92564574e-01 + 7.92437137e-01 + 7.91062711e-01 + 7.90950954e-01 + 7.86606332e-01 + 7.81219385e-01 + 7.80885444e-01 + 7.79688520e-01 + 7.77734693e-01 + 7.71636235e-01 + 7.67785706e-01 + 7.66342400e-01 + 7.62000669e-01 + 7.61677896e-01 + 7.59151323e-01 + 7.58474241e-01 + 7.55677717e-01 + 7.51517105e-01 + 7.45390836e-01 + 7.15538634e-01 + 6.87978167e-01 + 6.70584882e-01 + 1.00000000e+00 + 1.41421356e+00 + 1.60778186e+00 + 1.73205081e+00 + 1.82261573e+00 + 1.89339972e+00 + 1.95124445e+00 + 2.00000000e+00 + 2.04203942e+00 + 2.07892474e+00 + 2.11173664e+00 + 2.14125255e+00 + 2.16804975e+00 + 2.19256811e+00 + 2.21515024e+00 + 2.23606798e+00 + 2.25554048e+00 + 2.27374691e+00 + 2.29083555e+00 + 2.30693045e+00 + 2.32213639e+00 + 2.33654266e+00 + 2.35022594e+00 + 2.36325253e+00 + 2.37568015e+00 + 2.38755936e+00 + 2.39893466e+00 + 2.40984541e+00 + 2.42032663e+00 + 2.43040955e+00 + 2.44012219e+00 + 2.44948974e+00 + 2.45853495e+00 + 2.46727843e+00 + 2.47573888e+00 + 2.48393337e+00 + 2.49187748e+00 + 2.49958547e+00 + 2.50707045e+00 + 2.51434447e+00 + 2.52141865e+00 + 2.52830327e+00 + 2.53500784e+00 + 2.54154119e+00 + 2.54791152e+00 + 2.55412646e+00 + 2.56019313e+00 + 2.56611818e+00 + 2.57190782e+00 + 2.57756788e+00 + 2.58310382e+00 + 2.58852076e+00 + 2.59382352e+00 + 2.59901664e+00 + 2.60410440e+00 + 2.60909082e+00 + 2.61397973e+00 + 2.61877471e+00 + 2.62347919e+00 + 2.62809638e+00 + 2.63262936e+00 + 2.63708102e+00 + 2.64145413e+00 + 2.64575131e+00 + 1.33906683e+01 + 3.20562174e-01 + 1.35072710e-01 + 4.07098113e-02 + 2.11794076e+00 + + +0x6f4bcee8 +0xa120351 +0x138465cb +0x432b245e +0x84b5c806 +0x997e7258 +0x227f388b +0x28e3ab1b +0x421d3bbf +0x442ba1b6 +0x64391d8b +0x6c2941d8 +0x7d57e4d6 +0x99586fb1 +0xa94d1b8b +0xac6dc0d3 +0xc26a12d +0x17373663 +0x3b510d68 +0x4ac8890a +0x6d723b2f +0x7b34b44b +0x93bfccf7 +0xacd24172 +0xd02c2d3f +0xe1233d9a +0xee32095 +0x5303255d +0x712d751a +0x7cc78cda +0x95f1d4f6 +0xc01c87b3 +0xc07f3ffa +0xc72a388b +0xe2929c6a +0xeb348937 +0x5847412 +0x2c071f5e +0xa31de36c +0xdd1725b4 +0x6d62056 +0x206a9b13 +0x357fb6dc +0x40403451 +0x7e6ff0ed +0x95df6379 +0xf40d85f7 +0x2dc8fa +0x84a03b6 +0x1d85d299 +0x43317bee +0x5acd23ed +0x5cd118b9 +0x6eba4915 +0x7b7da752 +0x7f1b1898 +0x8739361b +0x9a638de8 +0xa68fcac0 +0xbb556c84 +0xbe2d70d7 +0xfcf5a2d +0x37cdbf33 +0x3b8bcc12 +0x3e94b2ce +0x60758ba6 +0xcaf8696d +0xcc65bb8f +0xcf8ccf4d +0xd83f8e6b +0xde199f1a +0xf81f3d71 +0xf9f1cb56 +0x13571c34 +0x24e4cf6b +0x6001d2d7 +0x7af9610d +0x92c1c729 +0xd5bdaaec +0x85ff9f1 +0xb2af6bbf +0xc57429bf +0x8ed28fa +0x1710a8dc +0x1f1f3c5b +0x28e138e0 +0x2cba622e +0x518fd0a4 +0x97d94dfb +0x99304a6c +0xf4ab0ca5 +0x29f04e8a +0x2b6d27f9 +0x2d1a3939 +0x53c6f488 +0x568217be +0x5f004c1c +0x7c978925 +0xa747f629 +0xba582bb6 +0xc1eb1afc +0xc6b1ef53 +0xdae2220a +0xdda84672 +0xf2bd8da5 +0xfe8da725 +0x54bca668 +0x55643e33 +0x59e721ba +0x64d23b47 +0x76efb8c6 +0xd2eb572f +0xdd94f48b +0xe30fedef +0xec14de0c +0x6080c +0xa417c32 +0x1e3600b1 +0x2adde3bc +0x46b00f60 +0x571d0e79 +0x65d5ba44 +0x7d457067 +0xb97a92d2 +0xe401c070 +0xef18b9e8 +0xfbcd2871 +0xfcf1ce4e +0x2a36077f +0x584d9f5a +0x7310315a +0x75860f31 +0xb83a8fae +0xe15ea0d3 +0xf1cb335b +0x399ec435 +0x66f977f8 +0x6fc2ae12 +0x90da81d3 +0xae4338b2 +0x2fd11eef +0x314e3275 +0x339be4de +0x4bb1a85a +0x696076dc +0x6a75e545 +0x74a673c2 +0x95c3f6c4 +0x96ebf11b +0x9e81544d +0xcb531ce1 +0xe0fb9e39 +0xeb533634 +0x2d5f79f3 +0x5201b63b +0x70df4399 +0x71fbcec0 +0x8b25428f +0x8b7ecdd2 +0xc79da0a1 +0xd9e7792d +0xe0590751 +0xe520b673 +0x5142fad3 +0x560edce2 +0x889f1d83 +0xa5813bc0 +0xbfcef464 +0xc69b1eb4 +0xcce7fa8e +0x5bd32969 +0x69561a5d +0x76ca0795 +0x95aba5ec +0xad66db78 +0xf28c3a41 +0x1670a74c +0x6d979318 +0x70054b29 +0x71984aa0 +0x73fe75e7 +0x746d96bd +0x7ef1efda +0xbadbd50c +0xd3a9e306 +0xddb91faf +0x1d4c9fe0 +0x4f6e7873 +0x523ca5c7 +0x72795795 +0x882071e3 +0x94847e33 +0xdd828c03 +0xe04cb029 +0x228be8b +0x1a820348 +0x1cf563cc +0x5aec6e8f +0x5e2547a9 +0x6c823adb +0xa1a81082 +0xdb85e3e8 +0xeea708a3 +0x779e5e2 +0x11e756cf +0x187fa052 +0x29a5e530 +0x38ab4334 +0x62d7d1b9 +0x770c0f3a +0x79f63501 +0xab399575 +0xab6485ad +0xcfbea9e8 +0xdc2e70c1 +0x205465e1 +0x302c665f +0x5dbd6fec +0x6b1028d1 +0xdd0d8204 +0xde5e1e93 +0x337ec0f +0xec6c409 +0x7f93c23c +0xa551f8ad +0xb83ec03b +0x1034ad32 +0x2c3bffce +0x2ce378ae +0x2da749b1 +0x4cac9c1c +0x6cecdfbc +0x795ad09a +0xa9948510 +0xd749bc20 +0x71c8d20e +0x953b3138 +0x9a44eb4e +0x9be3ab0c +0xbdbc6621 +0xbff92573 +0x922c4f3 +0x4cb9c7d1 +0x571045d4 +0x65a14628 +0x9adaafa8 +0xf13c8cbe +0x4c0c3c6 +0x25825f76 +0x2d1b26fc +0x39005781 +0x550c3dd3 +0x57440627 +0x71b49971 +0x7aeb5823 +0x7f9eab67 +0x8d3da149 +0x9996792d +0xa0b7a040 +0xbf50a3c5 +0xc17b75b0 +0xc36cdec6 +0xea6adde9 +0xf80ac34b +0xf9800514 +0xfde83a1b +0x7262b86 +0x2ee76723 +0x800fa3e2 +0x8e517671 +0xbacff499 +0xd2555d46 +0xff3efc12 +0x36e4817 +0xa2ef1fbd +0xaa5150a4 +0xb3555ac9 +0xc8a55c7a +0xcf5c917a +0xdfd05260 +0x67fd80b +0x2defc373 +0x879da32c +0xf12acb26 +0xf49f56bd +0x3bd14b2 +0x2bdd1930 +0x39eb80dc +0x73e5496c +0x8c06c25b +0xdba2ceb3 +0xe4b1c9d5 +0x44f051b +0x35a30127 +0x38ed5bf7 +0x81ddb0d7 +0x9f462b9c +0xbbc8e818 +0x15231688 +0x4f1a06ec +0x57f0d0bf +0x626d650e +0x64971440 +0x6508f6db +0x2d6f46db +0x4edecb9b +0x52d10e98 +0x62b5f4b0 +0x8721785a +0x9622bc62 +0x9dcb784a +0xe43096c9 +0xfaadcb8d +0x4545ab9a +0x59e2186c +0x747f6bdd +0x844f2870 +0x992f0181 +0xca642b54 +0xe61bb21c +0xeb6302d6 +0xfcd44e5a +0x36e51bcd +0x65c7d387 +0x852c2591 +0x8ac62901 +0x9f117c1e +0xd8e1c945 +0xf66ff8ee +0x1234f294 +0x7006aec0 +0x73e37186 +0x1040cab0 +0xa35d867e +0xe4048ee2 +0xfee2374e +0x2a78a315 +0x484c51e3 +0x60873d86 +0xb04d0cd8 +0xdbab4b5f +0xe961b412 +0xf4952170 +0xfc9d39b8 +0x83dc0dd4 +0x845c3f1e +0x94985529 +0xcbe13821 +0xe4aab5cb +0x3677d91d +0x3bf1fd7c +0x4d69ba48 +0x503bb197 +0x699da624 +0x1eda37fb +0x3f2719f1 +0x579a3017 +0x65645ed0 +0x7ed1474e +0x9cf9c94d +0xeddf10f4 +0x23d5b17c +0x46fbde1c +0x780d0a02 +0x971f55ca +0xa39499b9 +0x5be7cd51 +0xbe1f8ceb +0xf2385f85 +0xff3fa26d +0x11e0a0f +0x2338c49f +0x3d67dd41 +0x503c5ec0 +0x5c549280 +0x7630eef6 +0xd41c1f55 +0x202abde6 +0x7021fcad +0x91d6e493 +0x9bd682e0 +0xb92e6f65 +0xf11a7333 +0x3bbfbbf +0x1f30b88f +0x4e606205 +0x95b9eb32 +0x98f590d7 +0xb6b0c0e5 +0xca0c1d39 +0xd691df +0x73637d0 +0x2046f71f +0x3b5d2033 +0x50f69be0 +0x657ba33d +0x9b03c68a +0xaa475806 +0xc3a02ff9 +0xc9cac584 +0x2ceb17da +0x6a0ddbbb +0x6ef835a3 +0x9a8a30fc +0x9c53d5dd +0xaaf06dbb +0xe638bbe3 +0xf97d3b10 +0xa93e9aa +0x7c5563df +0x92d20d3f +0xc677864d +0xab0566 +0x1e5b4c55 +0x2e3cbc4c +0x5dcf1ade +0x78131334 +0x8822eb05 +0xb5b520d0 +0xbd502e9b +0xca6e3157 +0xf7a9cdb5 +0xf2fc2d2 +0x1f3349e1 +0x261e0e57 +0x3d338bfe +0x37b85314 +0x43b2095f +0x45d95945 +0xbfde4a4f +0xd6bc7e4a +0xf7385b44 +0x8d68be6 +0x35d96e87 +0x3861fd3b +0xc7057340 +0xc8901a9f +0x75656739 +0xcd71edf2 +0xd81a826e +0xdaa2e10f +0xdc4851f4 +0xb1fd719 +0xc27a5d6 +0x1e09f7e5 +0x2935c795 +0x4a076b16 +0x4e4cd9ef +0x5228ea90 +0x6ec3f247 +0x2d688b7 +0x74356e3 +0x184c0559 +0x2ca4522d +0x2fbf7b32 +0x3aa703d1 +0x60b5a14e +0x7594275e +0x7e74561e +0x966fdb70 +0xda04361b +0x5efc24cc +0x96026939 +0xc170c542 +0xcc490fe3 +0x364baa48 +0x81692e4e +0xae58b399 +0xbad3550 +0x2b3417b5 +0x3bfb5a4e +0x6a815b8a +0x7965d209 +0x9c8d3b5e +0xb4a8d3ba +0xc9f6095e +0xcc479751 +0x352067b7 +0x42e8b450 +0x5edae4bf +0xd34f1166 +0xea56cbce +0x4b2edd87 +0x724b594c +0x97066a83 +0xb3772a33 +0xefe94cac +0xfa6a0ec5 +0xa8ea46c +0xc3462038 +0xc3dc7db0 +0xc688262e +0xe31c29c8 +0x88a905e2 +0xe0c7ab61 +0xb6a19782 +0xa3de2da2 +0xe1326f37 +0x40618840 +0x71546018 +0xf8af29b4 +0x82053870 +0x6340a61f +0xb57b12ac +0x2e71b59 +0xbb56654a +0xefa8f093 +0xf6e96ce8 +0x298a66e7 +0x2b4207dc +0x882a5b90 +0xa74d50d6 +0x5b61e8e8 +0x319fca34 +0xa6197e7c +0x122bba0f +0x1d69f5cb +0x38718bcb +0x44ac1b34 +0xb4fd4c6 +0x259f04dd +0xa2cc9f2b +0xe52ad868 +0x51b7c2bb +0x91004568 +0xe1bf9484 +0x11e5b502 +0x89634782 +0xab91754 +0x43a81a4 +0x7e56eeb1 +0x99f90715 +0xa37f07de +0xbb6e3c78 +0xd82b9f50 +0x7b28d399 +0x41e4222d +0x4af9a820 +0xd9c27be6 +0x654197cd +0x1fd839a8 +0x3081ff4a +0x731331a +0x35d95276 +0xed919554 +0xa04660d +0x444d407e +0xbd55a089 +0xe29dd4b8 +0x91ebc792 +0x1b4ce03d +0x1136bee1 +0xe761f307 +0x23c2c99e +0xe049b330 +0xeab649d2 +0xe326fd59 +0xf5af811d +0x8a881d71 +0x9e9268ff +0xebf54fca +0xb48113e6 +0x9d14def6 +0xb87c0d68 +0x9aa12c81 +0xe6404c5e +0xa56ee90a +0xa19ce659 +0x931fca +0xef7bb384 +0x8a914727 +0x8543e62e +0x2660dfff +0x25842b60 +0x551b6841 +0xad5a8118 +0xb0864a3e +0xbcbcd1ce +0x47b59399 +0x907e5334 +0xa09bfa67 +0x13d599c2 +0xa62c920c +0xec660d7b +0x1163643c +0xa3dafff3 +0x79f52c05 +0x1e4697bb +0x9d285a65 +0xd1354cc0 +0xf9f8781a +0x97b6080a +0x174e5ef9 +0x89ebb84f +0x8a373ac1 +0x3764d05b +0x4e7f6fd7 +0xd3d5e28 +0x39cd8750 +0x630f928 +0x7611449f +0x9393e8a9 +0x727f3b86 +0x786d45b9 +0xb7458c39 +0x24aef43f +0xae00d1a2 +0xa3a90ad +0x9134cf02 +0x9696754d +0x2e32c1be +0x44b296e2 +0x1c2ecfb8 +0x3530770c +0xb45c06a9 +0x4dd308b5 +0x7243fc91 +0x50b99d42 +0xc7f04b79 +0xab43a647 +0x8c2da750 +0xa65b66eb +0x287915e3 +0x78807a32 +0xbc2aeb26 +0xaab91ef +0x1baffd3c +0x8609fe4e +0x495fadd4 +0xde5852a +0x827cf34e +0x53444b9f +0xbad34b5f +0xc35cdd8d +0x9907fa91 +0x57582909 +0xdef87cbe +0x6089358d +0x72910b14 +0x58840a79 +0xb9017f4f +0xdb20f1f +0x2d3768ea +0x5743ba9c +0x7d039184 +0xd8c61975 +0x33ffa14 +0x6ee9511b +0x2a5700eb +0xc277034a +0x8c343615 +0xc3521ab +0xe7404d61 +0x5870b91 +0x71217873 +0x9fa21106 +0x9d72eaf9 +0x301aaec6 +0x70e07921 +0x17da9c44 +0xa8430147 +0xd634ab4 +0x40b7219b +0xa659b9a +0x3b14ef0f +0x469d503c +0x1c13ba29 +0x1b8ca411 +0x35c91877 +0x67a53ef7 +0x6b6b057e +0x155334 +0xf1925441 +0x6c4ee6cf +0x2a24b75e +0xdb6a0b4b +0xbb7b6508 +0x26a99286 +0x3b0363e +0x9d2f0c89 +0xe97eee9 +0x2f21fe1b +0xc79be60f +0x190f871e +0x983ec133 +0xe226b1f8 +0xab6c5360 +0xf2961dc8 +0x78255606 +0xac340315 +0x9424eb54 +0x19816417 +0x69fa1973 +0x72bb210 +0x9fc6f3c7 +0x879b51cc +0xa224cd3a +0xc9e23a5a +0x76773e7 +0x96cb2440 +0x890af497 +0xc203a723 +0x721689af +0x64ba6dea +0xdd6dda1c +0x26c4b02a +0x568905b8 +0x138b01b0 +0x19e01b15 +0xce98448b +0x67ac2759 +0xb7383114 +0xe123b802 +0x3c941323 +0x2ad3d771 +0x84985f18 +0xe6e7f6c3 +0xf2c128f4 +0x4b7f1590 +0x6745d7a8 +0xfa08adde +0xf39edbd0 +0x5d13b978 +0xd74c73ba +0x95a13403 +0x1eb19fcc +0x61a949da +0xab4ba105 +0x400239d8 +0xf23cb05f +0x4444e02b +0xc4165ad +0x172d0010 +0x192244c4 +0x91ed43e4 +0x4510aec0 +0x6f9a01a4 +0xcab0cdbf +0xda439fbf +0x2a1dacc9 +0x85e3d355 +0xbec6f9a7 +0x149a7566 +0x91c2a8f1 +0xdc78512a +0x5ab2e762 +0xbe2709dc +0x49024bb0 +0x3b0ea898 +0x7dbd73df +0xa399ec70 +0x25cf4dd7 +0x29a42a00 +0x4bcea0fa +0x8c2a4e34 +0x651ab17e +0xc005f151 +0xc93c065e +0x31b192e5 +0x2b1d3501 +0x8ff893a1 +0x2cb26948 +0x5e70d986 +0x931bab17 +0x7a6cac2b +0x977f4573 +0x1acfc672 +0xb3ebcdda +0xd887ff7 +0x34b462f1 +0xaf0f1606 +0x2b290a25 +0x6b2e56ad +0x89b13aa8 +0x3c4e86 +0xd48fcef9 +0x37ac668e +0x7da732e8 +0x51622f88 +0x974d2152 +0x51872616 +0x7b68c846 +0xdbf3ec51 +0x4be2cf2f +0xd675c2b5 +0x1c920449 +0x8d5f9cb5 +0x87f48326 +0xb54dee68 +0xdaad65fc +0x5e9bac80 +0x803ea481 +0x6bf8a6f +0x662443f6 +0xe87fc210 +0x603e58ce +0xce72b27 +0xdac54173 +0x2acf49f3 +0x79ca8d86 +0x1efc7261 +0x52d69cdc +0x8a048546 +0x1b7b640f +0xf26531c6 +0x54f943ad +0x91cea369 +0xc083a40d +0x9ea45189 +0xaa4a6448 +0x440c37a5 +0xc0b9d8d9 +0xedd311c5 +0x5118b552 +0xaf0c013d +0x6c153fff +0x9263e559 +0xf31a5c6 +0x567fa55d +0xce5181f8 +0x1fbdafa8 +0x11e0ef8e +0x2cfe3012 +0x2d00064e +0x7367c807 +0x6af03075 +0xb0acf09f +0xa5eef404 +0x50297ad8 +0x6617f864 +0x69095277 +0x65ad13a9 +0x25162677 +0x3f5b2651 +0xf6adc487 +0xab006cf9 +0xf02e570f +0x4dd50367 +0x7580848b +0x9dcea98e +0x49c193ed +0xe501077e +0xab3b5a1e +0x9179832e +0x1cb635e9 +0xbd767578 +0xf4be476e +0xa160dc98 +0x3086a7e5 +0x789f5d09 +0x2697ca1b +0x8bc22dc6 +0xfbb32e76 +0x5efa66a8 +0xc5e80876 +0x7242677b +0x57399dc6 +0x8d2110 +0xff4ba2be +0x58e282ec +0xf4c28427 +0xe7a09ccd +0xc8a828fb +0xc76cdd0 +0x5beecd50 +0x2354f192 +0x57549aa3 +0x758daff8 +0x3dde6cc8 +0xd0611cf2 +0xf7de7989 +0x548ec995 +0x1702a280 +0x6b1e723a +0x6c7b6bf4 +0x324c191a +0x200f0c98 +0xe15b27a1 +0x841a0b8b +0xc385c6bc +0x1c1e9975 +0xdf1a480 +0x14b49a4c +0x97033997 +0xcdfcb9 +0x7cb95596 +0xb03b3ab6 +0x72bd11e0 +0x1723fe0d +0x7b3fd108 +0x582bcf95 +0x8bc13632 +0x6b08fd3c +0xa92829a2 +0xe87a72e5 +0xcc644f7f +0x16b08754 +0x7c0182d0 +0x4a1351b9 +0x90f4397 +0x460c13a1 +0x45993b76 +0x5e5390b3 +0x5aab2d4 +0x5a34380b +0xf43cc96a +0x84dda883 +0xaffb7bc7 +0x7167e30b +0xba0a4e29 +0xc9245ac6 +0xbeca5133 +0x8675536d +0x3978dacb +0xb3318639 +0x4734e26f +0x8e6f56ac +0x676a549a +0x4efdcbe1 +0x6af6efb6 +0x48bed789 +0xb8474810 +0x63ef5730 +0x3b7c6344 +0x2a0ec857 +0xb3b076af +0x4236c51a +0x87ddda39 +0x9f04b44b +0x6f5fb086 +0xc9b9aec5 +0x700de653 +0x18ee18e4 +0x18b520 +0xb5b1530b +0xf25b3813 +0xaccc94cb +0xb2297622 +0x8d8169 +0xb6421702 +0x8fb7c908 +0x54bd1859 +0x515bf503 +0xed06a095 +0xb272c0f8 +0xe118e61e +0x8a85be47 +0x3b2e1e6d +0xf4a53b70 +0x96341fb8 +0x24f997f9 +0xa13a29c8 +0xea073853 +0xa009aa70 +0xb1323805 +0x3bb0e10b +0xc6fd9bef +0x7b324b4 +0x38f29a0d +0xf3005f6c +0x2ed705cd +0xb4e2b8dc +0x33a3a894 +0x5ddc860e +0xe3527c1a +0x3390dca8 +0xd2f300a6 +0xf3cc75a +0x81cd264a +0x2a314da1 +0xca0bb8a0 +0xee38ea04 +0xfcb00932 +0xd5574099 +0x32a68989 +0x268c8411 +0x9d0c03 +0x7affbece +0x335abae2 +0x6c12ba4d +0x7792336b +0x4db356cc +0xbe28207 +0x726fcb2f +0x133074af +0x90790e2c +0xe0346663 +0x4fec96ff +0x92eb9116 +0x7cde9f87 +0x2549058d +0xa65f0aca +0x36a72919 +0xdd9bf693 +0x9a2f57b3 +0x2b8ac742 +0x4ffb1ded +0x52f765fa +0x967ef100 +0x545c6155 + + diff --git a/Ghidra/Features/BSim/data/lshweights_64.xml b/Ghidra/Features/BSim/data/lshweights_64.xml new file mode 100755 index 0000000000..9e721e2524 --- /dev/null +++ b/Ghidra/Features/BSim/data/lshweights_64.xml @@ -0,0 +1,1587 @@ + + + 1.00000000e+00 + 9.99459306e-01 + 9.98918319e-01 + 9.98377038e-01 + 9.97835465e-01 + 9.97293597e-01 + 9.96751434e-01 + 9.96208977e-01 + 9.95666224e-01 + 9.95123175e-01 + 9.94579829e-01 + 9.94036186e-01 + 9.93492246e-01 + 9.92948008e-01 + 9.92403472e-01 + 9.91858636e-01 + 9.91313501e-01 + 9.90768067e-01 + 9.90222331e-01 + 9.89676295e-01 + 9.89129957e-01 + 9.88583318e-01 + 9.88036376e-01 + 9.87489131e-01 + 9.86941583e-01 + 9.86393730e-01 + 9.85845573e-01 + 9.85297112e-01 + 9.84748344e-01 + 9.84199271e-01 + 9.83649892e-01 + 9.83100205e-01 + 9.82550211e-01 + 9.81999908e-01 + 9.81449298e-01 + 9.80898378e-01 + 9.80347148e-01 + 9.79795609e-01 + 9.79243758e-01 + 9.78691597e-01 + 9.78139124e-01 + 9.77586338e-01 + 9.77033240e-01 + 9.76479829e-01 + 9.75926104e-01 + 9.75372064e-01 + 9.74817710e-01 + 9.74263040e-01 + 9.73708054e-01 + 9.73152752e-01 + 9.72597132e-01 + 9.72041195e-01 + 9.71523988e-01 + 9.70928366e-01 + 9.70371473e-01 + 9.69814261e-01 + 9.69256727e-01 + 9.68698873e-01 + 9.68140698e-01 + 9.67582200e-01 + 9.67023380e-01 + 9.66464237e-01 + 9.65904770e-01 + 9.65344979e-01 + 9.64784864e-01 + 9.64224422e-01 + 9.63663655e-01 + 9.63102561e-01 + 9.62541141e-01 + 9.61979392e-01 + 9.61417316e-01 + 9.61583161e-01 + 9.61447834e-01 + 9.61380287e-01 + 9.61312818e-01 + 9.61245425e-01 + 9.60842679e-01 + 9.60775821e-01 + 9.60575700e-01 + 9.60509144e-01 + 9.60442663e-01 + 9.60376256e-01 + 9.60309924e-01 + 9.60243666e-01 + 9.60177483e-01 + 9.59979377e-01 + 9.59913489e-01 + 9.59781934e-01 + 9.59519698e-01 + 9.59454320e-01 + 9.59323782e-01 + 9.59258620e-01 + 9.59193531e-01 + 9.59128513e-01 + 9.59063566e-01 + 9.58998691e-01 + 9.58869154e-01 + 9.58804491e-01 + 9.58739899e-01 + 9.58675378e-01 + 9.58610927e-01 + 9.57970248e-01 + 9.57842940e-01 + 9.57715905e-01 + 9.57652490e-01 + 9.57462650e-01 + 9.57273417e-01 + 9.57147597e-01 + 9.57022043e-01 + 9.56771730e-01 + 9.56584686e-01 + 9.56522470e-01 + 9.56336209e-01 + 9.56150532e-01 + 9.56088769e-01 + 9.55965435e-01 + 9.55903863e-01 + 9.55780912e-01 + 9.55719532e-01 + 9.55109196e-01 + 9.55048506e-01 + 9.54866806e-01 + 9.54745982e-01 + 9.54685661e-01 + 9.54625402e-01 + 9.54565204e-01 + 9.54505066e-01 + 9.54444990e-01 + 9.54384974e-01 + 9.54265124e-01 + 9.53907015e-01 + 9.53788123e-01 + 9.53669468e-01 + 9.53610230e-01 + 9.53432866e-01 + 9.53314917e-01 + 9.53079717e-01 + 9.52903925e-01 + 9.52845442e-01 + 9.52787017e-01 + 9.52379639e-01 + 9.52148096e-01 + 9.52032661e-01 + 9.51859927e-01 + 9.51745047e-01 + 9.51401732e-01 + 9.51344704e-01 + 9.51173947e-01 + 9.51060380e-01 + 9.50947029e-01 + 9.50833893e-01 + 9.50777405e-01 + 9.50271403e-01 + 9.50103684e-01 + 9.50047882e-01 + 9.49936435e-01 + 9.49658721e-01 + 9.48997424e-01 + 9.48778607e-01 + 9.48560586e-01 + 9.48451873e-01 + 9.48289172e-01 + 9.48018983e-01 + 9.47803705e-01 + 9.47589199e-01 + 9.47535692e-01 + 9.47428821e-01 + 9.47056267e-01 + 9.46738766e-01 + 9.46686012e-01 + 9.46475460e-01 + 9.46422937e-01 + 9.46161010e-01 + 9.46056558e-01 + 9.45952288e-01 + 9.45692401e-01 + 9.45588761e-01 + 9.45537007e-01 + 9.45278907e-01 + 9.44970643e-01 + 9.44714959e-01 + 9.44612988e-01 + 9.44460356e-01 + 9.44004770e-01 + 9.43803389e-01 + 9.43402637e-01 + 9.43253039e-01 + 9.43153514e-01 + 9.43103812e-01 + 9.43054152e-01 + 9.42658336e-01 + 9.42510573e-01 + 9.42363171e-01 + 9.42118298e-01 + 9.41971851e-01 + 9.41777138e-01 + 9.41680017e-01 + 9.41583052e-01 + 9.41293087e-01 + 9.41100549e-01 + 9.41052510e-01 + 9.40146952e-01 + 9.39628815e-01 + 9.39488272e-01 + 9.39254756e-01 + 9.39115077e-01 + 9.38975720e-01 + 9.38836684e-01 + 9.38790409e-01 + 9.38513500e-01 + 9.38375519e-01 + 9.38237851e-01 + 9.38100495e-01 + 9.37963450e-01 + 9.37599503e-01 + 9.37508857e-01 + 9.37192657e-01 + 9.37012708e-01 + 9.36833291e-01 + 9.36743781e-01 + 9.36699076e-01 + 9.36654403e-01 + 9.36076629e-01 + 9.35988226e-01 + 9.35811806e-01 + 9.35723787e-01 + 9.35591998e-01 + 9.35416722e-01 + 9.35285596e-01 + 9.35024188e-01 + 9.34720626e-01 + 9.34332542e-01 + 9.34203729e-01 + 9.34118004e-01 + 9.34075187e-01 + 9.33946917e-01 + 9.33098609e-01 + 9.32137423e-01 + 9.32095974e-01 + 9.31683021e-01 + 9.31518619e-01 + 9.31109537e-01 + 9.31028048e-01 + 9.30946668e-01 + 9.30743690e-01 + 9.30662687e-01 + 9.30541383e-01 + 9.30138765e-01 + 9.29978457e-01 + 9.29938445e-01 + 9.29778660e-01 + 9.29698923e-01 + 9.29579513e-01 + 9.29460335e-01 + 9.29064741e-01 + 9.28750094e-01 + 9.28710876e-01 + 9.28554256e-01 + 9.28125603e-01 + 9.28086782e-01 + 9.27970468e-01 + 9.27162383e-01 + 9.27047806e-01 + 9.26933442e-01 + 9.26591621e-01 + 9.26553758e-01 + 9.26478101e-01 + 9.26063648e-01 + 9.25689281e-01 + 9.25428572e-01 + 9.25168963e-01 + 9.25094990e-01 + 9.24836780e-01 + 9.24433196e-01 + 9.24214169e-01 + 9.23669979e-01 + 9.22595791e-01 + 9.22347813e-01 + 9.22030440e-01 + 9.21995276e-01 + 9.21819759e-01 + 9.21644739e-01 + 9.20023301e-01 + 9.19989260e-01 + 9.19955237e-01 + 9.19887247e-01 + 9.19144240e-01 + 9.18244364e-01 + 9.17881454e-01 + 9.17618845e-01 + 9.17422614e-01 + 9.17259559e-01 + 9.17032000e-01 + 9.16837609e-01 + 9.16676079e-01 + 9.16386378e-01 + 9.16161984e-01 + 9.16098019e-01 + 9.15398712e-01 + 9.15367111e-01 + 9.15209346e-01 + 9.15083421e-01 + 9.14895009e-01 + 9.14147001e-01 + 9.13807138e-01 + 9.13041530e-01 + 9.12677344e-01 + 9.12435726e-01 + 9.12015127e-01 + 9.11835732e-01 + 9.11746227e-01 + 9.11627086e-01 + 9.11182313e-01 + 9.10858143e-01 + 9.10770023e-01 + 9.10564890e-01 + 9.10477181e-01 + 9.09837701e-01 + 9.09118843e-01 + 9.08323369e-01 + 9.08154213e-01 + 9.08041694e-01 + 9.07733300e-01 + 9.07426411e-01 + 9.07093321e-01 + 9.07010323e-01 + 9.06761983e-01 + 9.06159019e-01 + 9.05697010e-01 + 9.05615829e-01 + 9.05534753e-01 + 9.05372912e-01 + 9.05050469e-01 + 9.04863136e-01 + 9.03829469e-01 + 9.02967836e-01 + 9.02915991e-01 + 9.02580030e-01 + 9.02425570e-01 + 9.02374167e-01 + 9.02117775e-01 + 9.02015508e-01 + 9.01964436e-01 + 9.01862416e-01 + 9.01811468e-01 + 9.01760560e-01 + 9.01608082e-01 + 9.01077278e-01 + 9.00951548e-01 + 9.00851143e-01 + 9.00826067e-01 + 9.00650808e-01 + 9.00177532e-01 + 9.00004052e-01 + 8.99683126e-01 + 8.99412833e-01 + 8.99046093e-01 + 8.98899988e-01 + 8.98222561e-01 + 8.97480864e-01 + 8.97385787e-01 + 8.97267140e-01 + 8.97148713e-01 + 8.96442732e-01 + 8.95698195e-01 + 8.95559555e-01 + 8.95375169e-01 + 8.95237227e-01 + 8.94916517e-01 + 8.94665653e-01 + 8.94393101e-01 + 8.93203020e-01 + 8.93025290e-01 + 8.92517044e-01 + 8.91188264e-01 + 8.90865683e-01 + 8.90204101e-01 + 8.90013296e-01 + 8.89213854e-01 + 8.89067665e-01 + 8.88797047e-01 + 8.88568947e-01 + 8.88238599e-01 + 8.87705366e-01 + 8.87562553e-01 + 8.87298164e-01 + 8.87075292e-01 + 8.86913686e-01 + 8.86091734e-01 + 8.84614665e-01 + 8.83705910e-01 + 8.83495049e-01 + 8.81553146e-01 + 8.80887872e-01 + 8.80302170e-01 + 8.80283951e-01 + 8.77922683e-01 + 8.76549263e-01 + 8.76103482e-01 + 8.75779626e-01 + 8.75322144e-01 + 8.75237774e-01 + 8.75153513e-01 + 8.74433296e-01 + 8.73065052e-01 + 8.72093389e-01 + 8.71916795e-01 + 8.71900765e-01 + 8.71183397e-01 + 8.70332731e-01 + 8.69415658e-01 + 8.69107651e-01 + 8.68831667e-01 + 8.66961073e-01 + 8.66206778e-01 + 8.65519150e-01 + 8.64867385e-01 + 8.64136357e-01 + 8.63188026e-01 + 8.62991560e-01 + 8.62963541e-01 + 8.61604814e-01 + 8.61112553e-01 + 8.60623890e-01 + 8.60569814e-01 + 8.60313548e-01 + 8.59523971e-01 + 8.59417623e-01 + 8.57492917e-01 + 8.57351087e-01 + 8.57248126e-01 + 8.57106811e-01 + 8.56544503e-01 + 8.55271257e-01 + 8.55071745e-01 + 8.54402720e-01 + 8.54181178e-01 + 8.52915204e-01 + 8.52506421e-01 + 8.51506951e-01 + 8.51377154e-01 + 8.45659801e-01 + 8.45163079e-01 + 8.44830357e-01 + 8.44627228e-01 + 8.44339606e-01 + 8.44127330e-01 + 8.43757425e-01 + 8.43715277e-01 + 8.43557453e-01 + 8.43305689e-01 + 8.43253355e-01 + 8.42898537e-01 + 8.42473098e-01 + 8.40656061e-01 + 8.39934517e-01 + 8.38738707e-01 + 8.36227824e-01 + 8.35578999e-01 + 8.35373400e-01 + 8.33024764e-01 + 8.32539338e-01 + 8.31613886e-01 + 8.30622102e-01 + 8.29859397e-01 + 8.29584105e-01 + 8.28629038e-01 + 8.27461596e-01 + 8.25831526e-01 + 8.25336872e-01 + 8.24998305e-01 + 8.20898157e-01 + 8.17527549e-01 + 8.17030108e-01 + 8.15671015e-01 + 8.15459884e-01 + 8.14941754e-01 + 8.14927803e-01 + 8.14371975e-01 + 8.13758664e-01 + 8.12466710e-01 + 8.08725067e-01 + 8.08482422e-01 + 8.07168564e-01 + 8.06762848e-01 + 8.04212889e-01 + 7.94908330e-01 + 7.93209390e-01 + 7.90192133e-01 + 7.85163202e-01 + 7.79389391e-01 + 7.78536038e-01 + 7.78290367e-01 + 7.73039113e-01 + 7.72901165e-01 + 7.71655290e-01 + 7.67714792e-01 + 7.67132106e-01 + 7.67106555e-01 + 7.65078903e-01 + 7.64227043e-01 + 7.64019926e-01 + 7.62379313e-01 + 7.61838972e-01 + 7.60450853e-01 + 7.55103005e-01 + 7.53955341e-01 + 7.38792006e-01 + 7.36783498e-01 + 7.24695418e-01 + 6.96098422e-01 + 6.68999073e-01 + 1.00000000e+00 + 1.41421356e+00 + 1.60778186e+00 + 1.73205081e+00 + 1.82261573e+00 + 1.89339972e+00 + 1.95124445e+00 + 2.00000000e+00 + 2.04203942e+00 + 2.07892474e+00 + 2.11173664e+00 + 2.14125255e+00 + 2.16804975e+00 + 2.19256811e+00 + 2.21515024e+00 + 2.23606798e+00 + 2.25554048e+00 + 2.27374691e+00 + 2.29083555e+00 + 2.30693045e+00 + 2.32213639e+00 + 2.33654266e+00 + 2.35022594e+00 + 2.36325253e+00 + 2.37568015e+00 + 2.38755936e+00 + 2.39893466e+00 + 2.40984541e+00 + 2.42032663e+00 + 2.43040955e+00 + 2.44012219e+00 + 2.44948974e+00 + 2.45853495e+00 + 2.46727843e+00 + 2.47573888e+00 + 2.48393337e+00 + 2.49187748e+00 + 2.49958547e+00 + 2.50707045e+00 + 2.51434447e+00 + 2.52141865e+00 + 2.52830327e+00 + 2.53500784e+00 + 2.54154119e+00 + 2.54791152e+00 + 2.55412646e+00 + 2.56019313e+00 + 2.56611818e+00 + 2.57190782e+00 + 2.57756788e+00 + 2.58310382e+00 + 2.58852076e+00 + 2.59382352e+00 + 2.59901664e+00 + 2.60410440e+00 + 2.60909082e+00 + 2.61397973e+00 + 2.61877471e+00 + 2.62347919e+00 + 2.62809638e+00 + 2.63262936e+00 + 2.63708102e+00 + 2.64145413e+00 + 2.64575131e+00 + 1.35049281e+01 + 2.02671876e-01 + 5.40692533e-01 + 5.19701356e-02 + 8.52635318e-01 + + +0x5448c6df +0x5e3fe72a +0x8732d39a +0xc530e221 +0x15231688 +0x4af9a820 +0x57b8eed1 +0x6458e44a +0x70dc3f72 +0x80db69c4 +0x883e80ad +0xaa2735e2 +0xb79561e8 +0xd70a0e6b +0xf8d7bb43 +0x18a90326 +0x26624a84 +0x64d47b96 +0x69ef7ced +0x7e74561e +0x8df7c0dc +0x8e0a2ade +0xbed61bfd +0xc67ba253 +0xe8bcdef0 +0xf5981b0b +0x45a2933 +0x13190a49 +0x215f1aa7 +0x38d38c7d +0x5f871d68 +0x76ef16fc +0x7aeb5823 +0x7b7750a3 +0x863beee2 +0x91bd3706 +0xa31de36c +0xb63b2eb0 +0xc04342d4 +0xce2c5f6a +0xe3a15bb5 +0x4769340 +0x1234f294 +0x523ca5c7 +0x5cd118b9 +0x6b96a43e +0x86c13e68 +0x22f8a934 +0x52cf9a8e +0x71834fb3 +0x83235fd2 +0x9cc68925 +0x9d32b8dd +0xa94d1b8b +0xda755845 +0xe7eace61 +0x50be46cf +0x59465595 +0x66704da1 +0x6c989e7b +0x72910b14 +0x7f747ad6 +0xb966b64e +0xcaed843f +0x2bc0b274 +0x481bf5b5 +0x53dd50af +0x66af2a17 +0x66effb61 +0x72ddd9c1 +0x8b7e5bda +0x9d59500e +0xa6712099 +0xc1b46bf8 +0xe09e52b2 +0x307a41e9 +0x4529106d +0x5f5fc02a +0x64057f77 +0x961b35cb +0xac980a2d +0xc32ccc3c +0xcb7e9475 +0xe283fd62 +0xf0a4c9c4 +0xf7865f6b +0xc26a12d +0x2269e38c +0x3b510d68 +0x5062a247 +0x94ea04a0 +0xb0b527cd +0xcd10963e +0xe36adb5b +0x378c407 +0x6762dc9 +0x3164751f +0x43f3acd9 +0x45cb971e +0x83bca74f +0x8609fe4e +0x8b2588f0 +0x99304a6c +0xa10d081f +0xa97c37d8 +0xad97f358 +0xb096b884 +0xb5e8fe45 +0xc61d40f1 +0xda05d88e +0xf2268826 +0xfcf1ce4e +0x44f051b +0x4c0c3c6 +0x814b706 +0x2a5553cb +0x651160f6 +0x687662fe +0x80c4600e +0x832696ed +0x838bd1dc +0x9c33d13b +0xbeb373dd +0xe1bf9484 +0x130b69d9 +0x7b5b46fe +0x95d8c0a1 +0xbb6e3c78 +0xc1419c9b +0xcc479751 +0xd746917e +0xf99232bb +0x3197e6b +0xa120351 +0x5b2143b4 +0x8652f68e +0xca72ab9b +0x2c0463f3 +0x3764d05b +0x47d7d2c0 +0x71546018 +0x84f3e421 +0xbe85fd1b +0x85ff9f1 +0x2ce378ae +0x5697cd59 +0x75773661 +0x851d4a3d +0xa01b265b +0xef18b9e8 +0xef7aa6c0 +0x4fdfbd17 +0xa1ef60b9 +0xb1b321cb +0xcd6e18ab +0xf8fed59a +0x140cd3a7 +0x22dc68a4 +0x23c1416d +0x5a9b2418 +0x945a820a +0xdb5554ef +0xe0fb9e39 +0xe7319ef3 +0x6484854 +0x61a59bb3 +0x68c82ef4 +0x9a4698e9 +0xc841989d +0xcbc3a84 +0x16fd97e0 +0x64caebdd +0x657ba33d +0x73be301b +0x7f9eab67 +0x7fc99e4d +0xa56cc22d +0xb115257b +0xb3506b41 +0xb97a92d2 +0xc100de24 +0x6bf8a6f +0x3d251994 +0x3f69c99e +0x41f051c4 +0x4f08515c +0x684d7b6d +0x953b3138 +0x99a46389 +0xcf5f6a25 +0xeff801cf +0xde9b85f +0x4edecb9b +0x9b48fa27 +0xd0459de4 +0xdd828c03 +0xeeddc83d +0x36878fdd +0xba4ea259 +0xeb43148b +0xee41e0ed +0xfc8c223e +0x206a9b13 +0x5b572f38 +0x6b1028d1 +0x8a4e1250 +0x954972b6 +0x9a8a30fc +0xb2b07cc7 +0xbbc8e818 +0xbd55a089 +0xce0670ba +0xf32256d5 +0xf736f83c +0x866a68c4 +0xa2a97d17 +0xac3e0fe2 +0xe6f53939 +0x1f24b8 +0x27c8abda +0x524a91f1 +0x5edae4bf +0x770c0f3a +0x9fbe774c +0xa1ed21ca +0x29ba56e2 +0x29d171cc +0x51c391b2 +0x8f4ef52a +0xa1f38b4d +0xa7b3557b +0xca7d57f9 +0x1e5b4c55 +0x1f05c1a7 +0x6468c434 +0x65ad13a9 +0x6eba4915 +0x77057ed9 +0x7de5502d +0x98df995b +0x9c4d5b1d +0xa700966e +0x50ea2ed +0x711e13be +0x78551ada +0x9fd2e615 +0xf81f3d71 +0x32025a20 +0x383e1d1b +0x6252fa50 +0x708b3bc6 +0x71b49971 +0x759850fb +0xa2ef1fbd +0xa52c8d53 +0xa5b9de59 +0xd6c6ed08 +0xf81759c3 +0xfc9d39b8 +0xfd2f1aa5 +0x1957649c +0x560edce2 +0x5705c30b +0x63064fa6 +0x6a09338b +0x6ce9bc21 +0x98f590d7 +0x9ea5eb80 +0xc88bc153 +0xcd71edf2 +0xd7c6f451 +0xfc2d6b34 +0xfc3dc4a6 +0x228be8b +0x699da624 +0x9af42190 +0x9da44d78 +0xb30123b6 +0xd6015337 +0xdd94f48b +0xec32c98d +0xef40b0ae +0xff3fa26d +0x56ad7868 +0x6812a2ef +0x83dc0dd4 +0xd9c27be6 +0xebba6482 +0x63afeafd +0x79f63501 +0xdaa2e10f +0xe56485b2 +0xddb591b +0x3f55d64e +0x770b188f +0x845c3f1e +0xb2f08204 +0xf7b1ee18 +0x1a26687b +0x466f1d74 +0x6a9c3465 +0x73e37186 +0x9bcee07e +0xb0c78243 +0xb48b9232 +0xc56f3af8 +0xf27884f7 +0x45c479fe +0x486fbc6d +0x967493cd +0xb84f8c64 +0xd041ebd5 +0xd4f20879 +0xdab00427 +0x19e01b15 +0x3413e302 +0x4a993497 +0x4d69ba48 +0x795ad09a +0x89acbe9e +0x9996792d +0xac0010d8 +0xd6bc7e4a +0x7ebe2e5a +0xba050516 +0xbfde4a4f +0x4221441a +0x7998ecb5 +0xca642b54 +0xd3c3dc1b +0xe638bbe3 +0x1c04cab +0x4f7ce84a +0x60b5a14e +0x7347bb29 +0xa72ef136 +0xf5d1086a +0xf5d58789 +0xf2fc2d2 +0x15c7c70e +0x16504ea7 +0x37b85314 +0x3ccc9053 +0x903d9acd +0xec03edb0 +0xf9d134a5 +0xd691df +0x2a9da9ae +0x65c7d387 +0x7242677b +0xa3b1bf37 +0xea6adde9 +0x47ec9c48 +0x68f93f9f +0x8635638b +0xa1d7c919 +0xbb56654a +0x631336af +0xa53212d3 +0xb2f9734d +0xbb19ca16 +0x31ddd40 +0x19f1815e +0x1f3349e1 +0x6cb8c449 +0xbff92573 +0xddde4258 +0xab0566 +0x323e3e07 +0x953b830f +0x9c9242ae +0xd671ff84 +0xeec33837 +0xf238f5e1 +0xfe035f1 +0x2046f71f +0x25ad078a +0x7abe5dab +0x81692e4e +0xb38dce70 +0xb84a5355 +0xbcc6e2be +0xbe5253fd +0xd8bf5a72 +0xe1477292 +0xeeb39349 +0xf979c313 +0x144914dc +0x2e3cbc4c +0x634770a6 +0x8a2f4c65 +0xab166c3b +0xbd502e9b +0xc28c9752 +0xdac24a7b +0xed152c6b +0x724b594c +0x7589d74e +0x88682595 +0x96026939 +0xba582bb6 +0xf7a9cdb5 +0xa1120c +0x11e953c9 +0x175e351d +0x238da226 +0x58419ab2 +0x716268e2 +0x8d612a2a +0xac4e13fa +0xb6a19782 +0xe53abca5 +0xeb533634 +0x49893d7e +0xad528e2e +0xbc16c181 +0xc69b1eb4 +0x6c823adb +0x4db1ccee +0x5d869056 +0xc0ae37b9 +0xf88d13f1 +0x54004833 +0x6d0468cc +0x85c3cf17 +0x92d20d3f +0x9c714a30 +0xc82a3bfa +0xd04c1855 +0xe4d7227d +0x64971440 +0x98fdcd13 +0xda04361b +0x3b0363e +0xe23c904 +0x44b296e2 +0x4bcea0fa +0xa2da6a1a +0xa9a3db90 +0x96bbb43 +0x13950e73 +0x57ad3f8b +0xa53462c5 +0xb5b520d0 +0x59e2186c +0x5bb8423c +0x9bd682e0 +0xc9ac6f46 +0xc9cac584 +0xe1326f37 +0xf91a0493 +0x12d0c933 +0x75656739 +0xd0d844e2 +0xde0e7862 +0xef847993 +0x24aef43f +0x352067b7 +0x95069e0f +0xa2cc9f2b +0xc6b1ef53 +0xc9a4edc0 +0xe5cf6f11 +0x1acfc672 +0xd46059ba +0xd74c73ba +0x6e547e44 +0x8a844cb8 +0xf13b664d +0x2b3417b5 +0x3bf1fd7c +0x54272f9b +0x584d9f5a +0x84b66eb2 +0xf52487a3 +0x75e584d1 +0x91c2a8f1 +0x9de75c06 +0xb87c0d68 +0xbd269243 +0xca6e3157 +0x10b88584 +0x25b3fe22 +0x6a815b8a +0xa10608f3 +0xe123b802 +0x4a076b16 +0x5142fad3 +0x65645ed0 +0x75860f31 +0x94985529 +0x9edb5a42 +0xc8c1a9de +0x14536662 +0x20ecdb06 +0x4dd308b5 +0x7cdd4ee8 +0x821d54eb +0x90180cf4 +0x9877f6cf +0xcc8a2ca2 +0xd0611cf2 +0xea997bbf +0xf20b15c4 +0x13d599c2 +0x387d75fc +0x79d95fcf +0x7a9e9189 +0x1583fcb3 +0x618d5e71 +0xb44d02cb +0x40b7219b +0xd2080d06 +0x841dc292 +0xdf2c7cae +0x3d403ee8 +0xe3bba022 +0x18e9371b +0x6249cfe3 +0xe1e0f9d2 +0x1cd8519f +0x3c6ff84a +0xe0a7ebdb +0xfa6a0ec5 +0x2b4207dc +0x1ffc10df +0x50316bb2 +0x67a53ef7 +0x2f8d1827 +0x6efc91ad +0x2e9b38a6 +0xf97d3b10 +0x50f69be0 +0x4852d33b +0x69ce1ae3 +0xf7012a26 +0xaa5ce607 +0xdd23c752 +0x7759cf36 +0xc5a2ae36 +0x2ab26267 +0x6aac89e0 +0xb0078e9b +0xed0e9355 +0x448ef2c3 +0xea56cbce +0x72180a94 +0xa99c7447 +0xbb7b6508 +0x661505c9 +0xf729762b +0x18be7c2a +0x8131fcea +0xa92fa794 +0x118fa377 +0x46a6fdc +0x155334 +0xc21cc8cc +0x7c2f7bc9 +0x89ebb84f +0x54165598 +0x8d24f683 +0xd63aec43 +0x4734e26f +0x6e770623 +0xad03dec2 +0xdb88b2c +0xb45c06a9 +0x2a9aff84 +0x4b86717d +0x3814be84 +0xa3dafff3 +0x727f3b86 +0x37ac668e +0x76e9ab90 +0xb7411936 +0xac489e57 +0x17da9c44 +0x5d5a644a +0x66a366ad +0x7b048910 +0x39cd8750 +0xf2ac2b9e +0x40320a45 +0xd8c61975 +0x78131334 +0x1b8ca411 +0xf2c128f4 +0xa34c1158 +0x192244c4 +0x92477550 +0x4c7e8ae7 +0x7bda30cc +0x83e147c0 +0xc73ff82c +0x88a905e2 +0xabc59afb +0x6af03075 +0x4ea49ecf +0x9e9268ff +0x268cf2ca +0x1b9e8e8 +0x5689a6f8 +0x9652a65 +0x87f48326 +0x3b14ef0f +0x35b8f2e5 +0x5714bfb1 +0x33ffa14 +0xaf363a0e +0x3908fd8c +0x12c8af3c +0x29a42a00 +0xa942e409 +0x44cd3d0b +0x96341fb8 +0xeb672de4 +0xf0220f43 +0x7e5013ba +0x121eb126 +0x50e3dee2 +0xb9cb4ba2 +0xde2a1745 +0x71217873 +0x129a31a8 +0x2cfe3012 +0x350af332 +0x29f327b3 +0x9aef10a3 +0x551b6841 +0x47f51f32 +0x1eb19fcc +0x4851bfb5 +0xd82b9f50 +0x7913ace1 +0x9eb75106 +0xf6aaaa6c +0x412fb94b +0x7b28d399 +0xfeb0f08c +0x73e700de +0x2a24b75e +0x78807a32 +0x4be2cf2f +0xc9e23a5a +0x630f928 +0x9eb241d1 +0x5ab2e762 +0xfc47c589 +0x9d72eaf9 +0xcccf5afc +0x53e7e91c +0xd23c9e3f +0x8e3cd8f5 +0xa7add15f +0xb9017f4f +0x7d039184 +0xc6c613f7 +0x3530770c +0x426d7f07 +0x5870b91 +0x58840a79 +0x17340bd1 +0xc3521ab +0x6af0cc2f +0xfd8302fd +0x967de71b +0xb61416a +0x1b7b640f +0xdb6a0b4b +0x69fa1973 +0xab4ba105 +0x49024bb0 +0x91cea369 +0x845c0ebb +0xa8430147 +0x2660dfff +0x269ce4d3 +0x3987e0b +0xa6fc7272 +0xbd767578 +0x11f725ae +0x4d85aa4e +0xd1588c17 +0xe6e7f6c3 +0x5419cc3 +0x1221400 +0xa659b9a +0xa34f377 +0xad4610b6 +0x60356614 +0xf67bf7f8 +0x1972fb7b +0xce98448b +0x61a949da +0x721689af +0x7c4be5b0 +0xc0d31c8e +0x1ebfcb6b +0xda439fbf +0x2b1d3501 +0x76773e7 +0x4b7f1590 +0xf70ab7dc +0xf2961dc8 +0x415a7341 +0x5e5cdaa6 +0x3f9f2d14 +0xf8330973 +0xec23b475 +0x9424eb54 +0xb800b2e9 +0x3c4e86 +0x682ce25d +0xa83a0fd +0xe501077e +0x4444e02b +0xcab0cdbf +0x4469e04e +0x2ad3d771 +0x7ae18572 +0x9609ba47 +0xf26531c6 +0x84985f18 +0x34b462f1 +0x9003a7ad +0x96221223 +0x261a5e18 +0x6f9a01a4 +0xa399ec70 +0xf9876840 +0xf23cb05f +0xc8901a9f +0x5e9bac80 +0x79d53a7e +0xb54dee68 +0x40316b77 +0x98d7e51d +0x1cb635e9 +0x5c22663a +0x7da732e8 +0xf25b3813 +0x66829fe0 +0x974d2152 +0x4703ac7b +0x8a560f5c +0x928ed697 +0xc3a577ee +0xc93c065e +0x98cd88d1 +0x7113c47a +0x26c4b02a +0x25d3ff63 +0x1971a009 +0x7d3b98dc +0xe710b0b +0x366d544a +0x93888784 +0x3dc94655 +0x3b0ea898 +0xe87fc210 +0xd5b0f4fc +0xa0ca42ce +0x58944e03 +0x7c7c8ad5 +0xdbf3ec51 +0x75d731ae +0xfa6b9c2f +0x76347594 +0x97b6080a +0xb5d30b80 +0x1e3ef33c +0xe72b3ecf +0xff043bc7 +0xc083a40d +0x7b68c846 +0x10225e3c +0x4c94c092 +0x9263e559 +0xf02e570f +0x568905b8 +0x5e70d986 +0xaf0c013d +0xc005f151 +0x603e58ce +0x44b1474c +0xa07465f +0xc70cdf51 +0xd6d6403c +0xb80a9fda +0x9ea45189 +0x48153e92 +0x263aeb15 +0xfb2af0a0 +0x3e29ca48 +0x1c920449 +0xc7ded3b5 +0xf49dfe36 +0x3c941323 +0x3086a7e5 +0xf360e2aa +0xe87a72e5 +0xec02778f +0x7580848b +0xe6d44b93 +0xf79bbf0d +0x138b01b0 +0x6edfa8a3 +0x4c05d6bf +0x3940f4b0 +0x803ea481 +0xaeabf4df +0xa224cd3a +0x3ee9c6f1 +0xbfdbf4b +0xa37f07de +0x4a5fb7e0 +0x31c3f7a2 +0x827cf34e +0x1fbdafa8 +0x41df42f7 +0x2697ca1b +0xbc3bd41 +0x227c0258 +0xffb26b +0x548ec995 +0xaa4a6448 +0x41ed8a70 +0xed785e28 +0xf4be476e +0x9695852a +0xd75f245f +0x2f4741e9 +0x7d0221b5 +0xb2e7d134 +0x545c6155 +0x23c069c2 +0x9179832e +0xd90d8982 +0x32ef95a0 +0x5a9d0a0f +0x6617f864 +0xa93e38de +0xdecced22 +0xf4c28427 +0xbb916bb7 +0x6a3233de +0x1723fe0d +0xc76cdd0 +0x49c193ed +0xe15b27a1 +0xe118e61e +0x7418d6c7 +0x5118b552 +0x7367c807 +0xe08cc249 +0xb3318639 +0xf37ee1d8 +0x49d8861c +0xb553e9a +0x758daff8 +0x3d0f6ca6 +0x2d5751aa +0xf3005f6c +0x1702a280 +0x36d3e2a6 +0x5beecd50 +0x678ec56b +0x48bed789 +0x845089c8 +0xf7de7989 +0x14b49a4c +0x2973f525 +0x6c7b6bf4 +0x1c3f00bd +0xb5baf42b +0x3978dacb +0xac340315 +0xe7a09ccd +0x3c7ae06d +0x5e5390b3 +0x9978d4ca +0xf82e31f +0xcdfcb9 +0x72bd11e0 +0x33dd5629 +0x90f4397 +0x3d9f361e +0x7167e30b +0x4f9d22c0 +0xa423706d +0x9a5d40e9 +0xc27fe8e7 +0x5aab2d4 +0xbeca5133 +0xd8007ae1 +0xba0a4e29 +0xc385c6bc +0x629bd0b0 +0x28e72823 +0x6af6efb6 +0xfa226145 +0x25c6b8b8 +0x5a34380b +0xf4600172 +0x4efdcbe1 +0xcd96b51e +0x6a0ddbbb +0xb5b1530b +0x8e6f56ac +0x63ef5730 +0x391d08e7 +0x200f0c98 +0x84dda883 +0xd953f05c +0x87ddda39 +0x3bb05ad1 +0x7f712fbb +0xe33a3e1f +0xeb1dfae4 +0x7c0182d0 +0x45993b76 +0xac34dfeb +0xc3f2355f +0x7b3fd108 +0xd5894d95 +0x8134ba1e +0x6f5fb086 +0xfa5ceeb2 +0xb6421702 +0x57399dc6 +0x335abae2 +0x6c12ba4d +0x7792336b +0x8f47cf00 +0xc2b52756 +0x54bd1859 +0x4cee5d20 +0xb272c0f8 +0x4a1351b9 +0x1390833b +0xf4a53b70 +0x8a85be47 +0x7a4c16d +0xb2297622 +0x2519a18b +0xea073853 +0xa13a29c8 +0x9d0c03 +0x2ed705cd +0x39990d69 +0x3a999609 +0x3d08206 +0xd5f83c8f +0xaccc94cb +0xb4e2b8dc +0x5197f553 +0x57549aa3 +0xef0c7d68 +0x58e282ec +0x5ddc860e +0xc8ff7f7a +0xf3cc75a +0xe3527c1a +0xcefef45d +0x27755936 +0x3390dca8 +0x11e0a0f +0x81cd264a +0xfcb00932 +0xca0bb8a0 +0xee38ea04 +0x37a6568d +0x72900e6f +0x7affbece +0x4db356cc +0xa8249670 +0x726fcb2f +0xe7428d49 +0x90790e2c +0xe0346663 +0xcd1eb1e5 +0x4fec96ff +0x6a07f21f +0x133074af +0x7cde9f87 +0x4dc6f6e1 +0x3d09de58 +0x4b2edd87 +0x2549058d +0xa65f0aca +0xdd9bf693 +0x9fa21106 +0xe6c82b47 +0x2b8ac742 +0x7adefe4e +0xd5574099 +0x52f765fa +0xc55041c4 +0xab6831d3 + + diff --git a/Ghidra/Features/BSim/data/lshweights_64_32.xml b/Ghidra/Features/BSim/data/lshweights_64_32.xml new file mode 100755 index 0000000000..b913a32bd2 --- /dev/null +++ b/Ghidra/Features/BSim/data/lshweights_64_32.xml @@ -0,0 +1,1587 @@ + + + 1.00000000e+00 + 9.99460904e-01 + 9.98921517e-01 + 9.98381839e-01 + 9.97841868e-01 + 9.97301606e-01 + 9.96761050e-01 + 9.96220201e-01 + 9.95679059e-01 + 9.95137622e-01 + 9.94595890e-01 + 9.94053863e-01 + 9.93511541e-01 + 9.92968922e-01 + 9.92426007e-01 + 9.91882794e-01 + 9.91339284e-01 + 9.90795475e-01 + 9.90251368e-01 + 9.89706962e-01 + 9.89162256e-01 + 9.88617251e-01 + 9.88071944e-01 + 9.87526336e-01 + 9.86980427e-01 + 9.86516873e-01 + 9.85887702e-01 + 9.85340885e-01 + 9.84793764e-01 + 9.84246339e-01 + 9.83698610e-01 + 9.83150575e-01 + 9.82602235e-01 + 9.82053588e-01 + 9.81504635e-01 + 9.80955375e-01 + 9.80405806e-01 + 9.79855930e-01 + 9.79305745e-01 + 9.78755251e-01 + 9.78204447e-01 + 9.77653332e-01 + 9.77101907e-01 + 9.76550170e-01 + 9.75998122e-01 + 9.75445761e-01 + 9.74893087e-01 + 9.74340099e-01 + 9.73786798e-01 + 9.73233182e-01 + 9.72679250e-01 + 9.72125004e-01 + 9.71731614e-01 + 9.71015561e-01 + 9.70460364e-01 + 9.69904849e-01 + 9.69349016e-01 + 9.68792864e-01 + 9.68236393e-01 + 9.67679601e-01 + 9.67122489e-01 + 9.66565056e-01 + 9.66007301e-01 + 9.65449224e-01 + 9.64890824e-01 + 9.64332101e-01 + 9.63773054e-01 + 9.63213683e-01 + 9.62653986e-01 + 9.62093964e-01 + 9.61533616e-01 + 9.60972940e-01 + 9.60411938e-01 + 9.59850608e-01 + 9.59288949e-01 + 9.58726961e-01 + 9.58343570e-01 + 9.57601996e-01 + 9.57039018e-01 + 9.56947095e-01 + 9.56895996e-01 + 9.56641157e-01 + 9.56488773e-01 + 9.56286199e-01 + 9.56235662e-01 + 9.56084309e-01 + 9.55983619e-01 + 9.55682564e-01 + 9.55482699e-01 + 9.55383017e-01 + 9.55184150e-01 + 9.55084964e-01 + 9.54788388e-01 + 9.54542360e-01 + 9.54493275e-01 + 9.54150805e-01 + 9.53907378e-01 + 9.53810283e-01 + 9.53713346e-01 + 9.53664936e-01 + 9.53568233e-01 + 9.53134990e-01 + 9.53087045e-01 + 9.52991269e-01 + 9.52800175e-01 + 9.52467219e-01 + 9.52088946e-01 + 9.51947706e-01 + 9.51900700e-01 + 9.51853731e-01 + 9.51713042e-01 + 9.51619433e-01 + 9.51386046e-01 + 9.51339477e-01 + 9.51060819e-01 + 9.51014501e-01 + 9.50737343e-01 + 9.50507349e-01 + 9.50323985e-01 + 9.50232513e-01 + 9.50186829e-01 + 9.49822598e-01 + 9.49596068e-01 + 9.49190450e-01 + 9.48832180e-01 + 9.48742944e-01 + 9.48698376e-01 + 9.48609338e-01 + 9.48476027e-01 + 9.48387316e-01 + 9.48343010e-01 + 9.48210286e-01 + 9.47463650e-01 + 9.47420017e-01 + 9.47376415e-01 + 9.47245798e-01 + 9.46855634e-01 + 9.46812438e-01 + 9.46639961e-01 + 9.46596918e-01 + 9.46510925e-01 + 9.46425054e-01 + 9.45954933e-01 + 9.45869847e-01 + 9.45446207e-01 + 9.45319692e-01 + 9.45235495e-01 + 9.45025513e-01 + 9.44983604e-01 + 9.44899873e-01 + 9.44234159e-01 + 9.44192794e-01 + 9.44110148e-01 + 9.43780687e-01 + 9.43534761e-01 + 9.43330582e-01 + 9.43249102e-01 + 9.43086471e-01 + 9.43045881e-01 + 9.42802910e-01 + 9.42641468e-01 + 9.42520668e-01 + 9.42480455e-01 + 9.42199704e-01 + 9.41920246e-01 + 9.41800870e-01 + 9.41761130e-01 + 9.41523235e-01 + 9.41207483e-01 + 9.41168129e-01 + 9.40893362e-01 + 9.40658832e-01 + 9.40386350e-01 + 9.40269944e-01 + 9.40192464e-01 + 9.39960614e-01 + 9.39806535e-01 + 9.39768077e-01 + 9.39232188e-01 + 9.38890159e-01 + 9.38814413e-01 + 9.38625457e-01 + 9.38062076e-01 + 9.37987350e-01 + 9.37689357e-01 + 9.37134509e-01 + 9.37060908e-01 + 9.37024141e-01 + 9.36804002e-01 + 9.36694230e-01 + 9.36584653e-01 + 9.36511712e-01 + 9.36329737e-01 + 9.36220811e-01 + 9.36039698e-01 + 9.35715039e-01 + 9.35213412e-01 + 9.35142086e-01 + 9.35106453e-01 + 9.35070842e-01 + 9.34680475e-01 + 9.34503854e-01 + 9.34292579e-01 + 9.33802414e-01 + 9.33663084e-01 + 9.33005549e-01 + 9.32765063e-01 + 9.32730785e-01 + 9.32457244e-01 + 9.31846192e-01 + 9.31442171e-01 + 9.31341579e-01 + 9.31207712e-01 + 9.31140887e-01 + 9.31040786e-01 + 9.30841070e-01 + 9.30575786e-01 + 9.30476599e-01 + 9.30377571e-01 + 9.30048622e-01 + 9.29950278e-01 + 9.29688794e-01 + 9.29656186e-01 + 9.29591023e-01 + 9.29493406e-01 + 9.28878728e-01 + 9.28333894e-01 + 9.28142742e-01 + 9.27604332e-01 + 9.27541297e-01 + 9.27258432e-01 + 9.26448458e-01 + 9.26263037e-01 + 9.26201353e-01 + 9.26139730e-01 + 9.26016669e-01 + 9.25648944e-01 + 9.24799311e-01 + 9.24739066e-01 + 9.24648808e-01 + 9.24348892e-01 + 9.24199477e-01 + 9.23990897e-01 + 9.23753375e-01 + 9.23634953e-01 + 9.23605383e-01 + 9.23575827e-01 + 9.23281037e-01 + 9.23104833e-01 + 9.22666493e-01 + 9.22346996e-01 + 9.22289082e-01 + 9.21369704e-01 + 9.21085159e-01 + 9.20717184e-01 + 9.20519939e-01 + 9.20351367e-01 + 9.20183249e-01 + 9.19987683e-01 + 9.19792728e-01 + 9.19709361e-01 + 9.18608317e-01 + 9.18553770e-01 + 9.18526515e-01 + 9.18472040e-01 + 9.18363233e-01 + 9.18227489e-01 + 9.18010910e-01 + 9.17956882e-01 + 9.17929886e-01 + 9.17687441e-01 + 9.17660560e-01 + 9.17258733e-01 + 9.17045475e-01 + 9.16859468e-01 + 9.16199629e-01 + 9.15624660e-01 + 9.15520684e-01 + 9.15416880e-01 + 9.15080699e-01 + 9.14823317e-01 + 9.14311695e-01 + 9.13652754e-01 + 9.11716346e-01 + 9.11667469e-01 + 9.11156533e-01 + 9.11035489e-01 + 9.10601643e-01 + 9.10577628e-01 + 9.10505637e-01 + 9.10409776e-01 + 9.10314060e-01 + 9.10099229e-01 + 9.10027781e-01 + 9.09956413e-01 + 9.09671746e-01 + 9.09035894e-01 + 9.08942234e-01 + 9.08732007e-01 + 9.07990137e-01 + 9.07921030e-01 + 9.07120304e-01 + 9.06802825e-01 + 9.06599566e-01 + 9.06531958e-01 + 9.05637327e-01 + 9.05570735e-01 + 9.05172642e-01 + 9.04864729e-01 + 9.04667569e-01 + 9.04492827e-01 + 9.03712374e-01 + 9.03069225e-01 + 9.02813772e-01 + 9.02305915e-01 + 9.02158549e-01 + 9.01927658e-01 + 9.01843905e-01 + 9.01802069e-01 + 9.01614148e-01 + 9.01012358e-01 + 9.00888554e-01 + 8.99987912e-01 + 8.99623091e-01 + 8.99280437e-01 + 8.99220159e-01 + 8.98600580e-01 + 8.98501206e-01 + 8.98481350e-01 + 8.98204007e-01 + 8.97809871e-01 + 8.97515850e-01 + 8.97437672e-01 + 8.97203706e-01 + 8.97125906e-01 + 8.97067618e-01 + 8.96912442e-01 + 8.96757639e-01 + 8.96718997e-01 + 8.95836574e-01 + 8.95665333e-01 + 8.95589372e-01 + 8.94928512e-01 + 8.94348788e-01 + 8.93755785e-01 + 8.93589980e-01 + 8.92967425e-01 + 8.92242611e-01 + 8.92062671e-01 + 8.91740036e-01 + 8.91294586e-01 + 8.90940421e-01 + 8.90905111e-01 + 8.90623321e-01 + 8.89976340e-01 + 8.89473723e-01 + 8.89387458e-01 + 8.89232468e-01 + 8.88923596e-01 + 8.88906479e-01 + 8.87971951e-01 + 8.87803472e-01 + 8.87484560e-01 + 8.87467819e-01 + 8.86818236e-01 + 8.86603142e-01 + 8.86520602e-01 + 8.86405222e-01 + 8.85473232e-01 + 8.85343486e-01 + 8.85311090e-01 + 8.84891399e-01 + 8.84650493e-01 + 8.84044146e-01 + 8.83822167e-01 + 8.83758883e-01 + 8.83632498e-01 + 8.83490606e-01 + 8.83396183e-01 + 8.83160719e-01 + 8.83082419e-01 + 8.82739009e-01 + 8.82289070e-01 + 8.82165491e-01 + 8.81919031e-01 + 8.81063650e-01 + 8.80609961e-01 + 8.80264245e-01 + 8.79756500e-01 + 8.79385647e-01 + 8.79282182e-01 + 8.79149395e-01 + 8.78752635e-01 + 8.78227318e-01 + 8.77922808e-01 + 8.77089026e-01 + 8.76406889e-01 + 8.75815751e-01 + 8.75452444e-01 + 8.75382806e-01 + 8.75021867e-01 + 8.74814526e-01 + 8.72977135e-01 + 8.72203128e-01 + 8.72123571e-01 + 8.71964744e-01 + 8.71806296e-01 + 8.70590979e-01 + 8.70500397e-01 + 8.70255152e-01 + 8.69780149e-01 + 8.69308513e-01 + 8.68928542e-01 + 8.68062748e-01 + 8.68000444e-01 + 8.67838725e-01 + 8.67060516e-01 + 8.64494580e-01 + 8.63976565e-01 + 8.63859391e-01 + 8.63544038e-01 + 8.61331975e-01 + 8.61320701e-01 + 8.58939572e-01 + 8.58743983e-01 + 8.58624735e-01 + 8.56860810e-01 + 8.55948927e-01 + 8.55296174e-01 + 8.51078346e-01 + 8.50480093e-01 + 8.50077782e-01 + 8.48959992e-01 + 8.46269920e-01 + 8.46035429e-01 + 8.46017425e-01 + 8.45846615e-01 + 8.45256555e-01 + 8.44574487e-01 + 8.44126421e-01 + 8.43342754e-01 + 8.42345654e-01 + 8.40576375e-01 + 8.40070809e-01 + 8.39225434e-01 + 8.38333840e-01 + 8.37844302e-01 + 8.35439274e-01 + 8.32905531e-01 + 8.31877540e-01 + 8.28951373e-01 + 8.28516121e-01 + 8.28467149e-01 + 8.26869853e-01 + 8.24338217e-01 + 8.23453243e-01 + 8.22418489e-01 + 8.22027926e-01 + 8.21417584e-01 + 8.21411254e-01 + 8.21221608e-01 + 8.20449303e-01 + 8.19883266e-01 + 8.18266112e-01 + 8.16524980e-01 + 8.16465894e-01 + 8.15643762e-01 + 8.15062242e-01 + 8.14072919e-01 + 8.12911298e-01 + 8.12776521e-01 + 8.11481007e-01 + 8.10888314e-01 + 8.09846501e-01 + 8.09250981e-01 + 8.09122861e-01 + 8.08941747e-01 + 8.06890819e-01 + 8.05168458e-01 + 8.04670134e-01 + 8.04529810e-01 + 8.04309845e-01 + 8.04264934e-01 + 8.02044221e-01 + 7.97927742e-01 + 7.95316410e-01 + 7.92122943e-01 + 7.86605727e-01 + 7.86217880e-01 + 7.83155249e-01 + 7.79109320e-01 + 7.74331417e-01 + 7.71262080e-01 + 7.68428340e-01 + 7.65248976e-01 + 7.61947424e-01 + 7.60628213e-01 + 7.55741923e-01 + 7.48121393e-01 + 7.47705213e-01 + 7.46952131e-01 + 7.43247170e-01 + 7.39923058e-01 + 7.38093043e-01 + 7.35576241e-01 + 7.31852812e-01 + 7.26474493e-01 + 7.22323410e-01 + 7.21355505e-01 + 6.92632792e-01 + 6.87113851e-01 + 6.70218118e-01 + 1.00000000e+00 + 1.41421356e+00 + 1.60778186e+00 + 1.73205081e+00 + 1.82261573e+00 + 1.89339972e+00 + 1.95124445e+00 + 2.00000000e+00 + 2.04203942e+00 + 2.07892474e+00 + 2.11173664e+00 + 2.14125255e+00 + 2.16804975e+00 + 2.19256811e+00 + 2.21515024e+00 + 2.23606798e+00 + 2.25554048e+00 + 2.27374691e+00 + 2.29083555e+00 + 2.30693045e+00 + 2.32213639e+00 + 2.33654266e+00 + 2.35022594e+00 + 2.36325253e+00 + 2.37568015e+00 + 2.38755936e+00 + 2.39893466e+00 + 2.40984541e+00 + 2.42032663e+00 + 2.43040955e+00 + 2.44012219e+00 + 2.44948974e+00 + 2.45853495e+00 + 2.46727843e+00 + 2.47573888e+00 + 2.48393337e+00 + 2.49187748e+00 + 2.49958547e+00 + 2.50707045e+00 + 2.51434447e+00 + 2.52141865e+00 + 2.52830327e+00 + 2.53500784e+00 + 2.54154119e+00 + 2.54791152e+00 + 2.55412646e+00 + 2.56019313e+00 + 2.56611818e+00 + 2.57190782e+00 + 2.57756788e+00 + 2.58310382e+00 + 2.58852076e+00 + 2.59382352e+00 + 2.59901664e+00 + 2.60410440e+00 + 2.60909082e+00 + 2.61397973e+00 + 2.61877471e+00 + 2.62347919e+00 + 2.62809638e+00 + 2.63262936e+00 + 2.63708102e+00 + 2.64145413e+00 + 2.64575131e+00 + 1.34172324e+01 + 2.30811100e-01 + 2.06166959e+01 + 9.61110575e-04 + 3.83193896e+01 + + +0x26bd6d43 +0x3ea116d8 +0x94985529 +0xe97eee9 +0x34ec8454 +0x52ee7a95 +0x5fb0bebb +0xa4246f03 +0xc8df51af +0xdded0b8e +0xefa3937a +0x184c0559 +0x49565c94 +0x5e70d986 +0x6b29d047 +0x84157a8d +0x8ac62901 +0x8ac8b8d9 +0xff3fa26d +0x2750177e +0x54344f26 +0x5b917c9d +0x71834fb3 +0x7f73a2a7 +0x90140cf0 +0x9ae34346 +0xb6d97963 +0xb83a8fae +0xd23e30d5 +0xd64b74f1 +0xe5c0138d +0xed308a6a +0xf904bece +0xfaadcb8d +0xfbeb753a +0x433bd3c +0x281b75b2 +0x3e191ca9 +0x524f2032 +0x7243fc91 +0x79955e95 +0xaf74a431 +0xdf74a6f3 +0x137ce8c2 +0x15ac38fa +0x1a58abde +0x28ec25d9 +0x31d4bcd1 +0x4fc67aad +0x6478941c +0x91d6ae1a +0xb083760b +0xe0930f5c +0xa1120c +0x1430577e +0x15d52b83 +0x1af73546 +0x2139ee8c +0x2fbf7b32 +0x3e53f63d +0xaa644a6d +0xdce664c4 +0x164fb77e +0x2fd11eef +0x3861fd3b +0x593cf505 +0x66f977f8 +0x828d8ea4 +0x9996792d +0xa0758a85 +0xb206b5cc +0xbb8e35b5 +0xc13d2e75 +0xd6eff4bf +0xef2b475d +0x1b4ce03d +0x4fdd5245 +0xa4fc7050 +0xbcc27ea1 +0xdb6a0b4b +0xddeb2aae +0x529fbe4 +0x2b6c216e +0x52a55d78 +0x918580a7 +0x91a87079 +0x9f117c1e +0xa61d2a74 +0x23d5b17c +0x31af2161 +0x35964cf1 +0x52d10e98 +0x58920e82 +0x63987283 +0xa1b3a460 +0xbaf74b87 +0xbf06d903 +0xc1419c9b +0xe6791c70 +0xf51baeab +0x11304194 +0x1acfc672 +0x5be7cd51 +0x7a08c358 +0x84ca749a +0x96cb2440 +0xa9f1efdf +0xb3ce0343 +0xd142b432 +0xe4aab5cb +0x50ea2ed +0x10f4e77f +0x1dbb14c8 +0x26c520c8 +0x66effb61 +0x69c2f5ab +0xa719086a +0xc1eb1afc +0xd186b073 +0xd675bcbe +0xe9fb08f1 +0xfacd9d3e +0x7bafc080 +0xa218a567 +0xb3ab651c +0x2e0e3896 +0x33181322 +0x36e51bcd +0x532b0e98 +0xb97a92d2 +0xbb6e3c78 +0xdc367001 +0xdcb56193 +0xfbbff370 +0x2d258260 +0x4c7b7cd9 +0x69e1656d +0x73e37186 +0x7891fd74 +0x7900d938 +0x85a3dba8 +0xcaed843f +0xd44ce35b +0xe04d3095 +0x3fb37a4 +0x67c5b83 +0x168b919f +0x1f30b88f +0x352067b7 +0x438f0850 +0x6bc875f4 +0x71cedc01 +0x7d039184 +0x90841844 +0x90da81d3 +0xc4afa95d +0xd4973b5f +0xd5032981 +0x3a72db3a +0x654197cd +0x9be3ab0c +0xad97f358 +0xb00eb120 +0xb5be1507 +0xab91754 +0x2ab26267 +0x2b6d27f9 +0x38ed5bf7 +0x88a905e2 +0x964617d9 +0x98e89567 +0xb77b955d +0xd84f53c6 +0xe4353964 +0x1a30aa8f +0x3e9aff40 +0x4b0eaded +0x55e674e4 +0x6812a2ef +0x7cdfc2e5 +0x8dc49302 +0xbb5600bc +0xc05983f8 +0xd0c57162 +0x205465e1 +0x442ba1b6 +0xa7e086ad +0xb23d5aa1 +0xeaec8a22 +0xcde5985 +0x1aecf5a7 +0x32908c60 +0x36316405 +0x392ad095 +0x3ac26fc0 +0x55f1287d +0x5da55624 +0x91ebc792 +0xdc3ecce0 +0xe432ce50 +0x172d0010 +0x26968db1 +0x309c7530 +0x4cc2afd6 +0x6eb2a42f +0x7b7da752 +0x8f3d68f3 +0xb54579bc +0xfee2374e +0x62e36596 +0x7ba7569e +0xb10ec6e4 +0xbcb8462a +0xec4a6e8c +0xf1d7bf7e +0x8d38c0d4 +0xd34f1166 +0xfbf360b3 +0xfe27c847 +0xf6f4020b +0x5bda8c5d +0x61a949da +0x85c5dc1f +0x8af9e979 +0xb12eb7bc +0xb7383114 +0xb7a154ea +0xc83206c0 +0xef7aa6c0 +0x2e3cbc4c +0x314e3275 +0x6b96a43e +0x79f63501 +0x9e5f84bc +0xb1b321cb +0x35852ac4 +0x49893d7e +0x5ee3db3d +0xbcbcd1ce +0x1f0b6597 +0x3552236e +0x7858c1fb +0xa1df469d +0xf7385b44 +0x12bc928d +0x2ca4522d +0x3178800a +0x6861a331 +0x7018608c +0x73a7059b +0xc362ce89 +0xd1b74f1d +0x548670d9 +0x6d17b95c +0xaa5a0967 +0xad66db78 +0xd5fb5acf +0xf97d3b10 +0x40618840 +0x50c998bf +0x548ec995 +0x5ad0b785 +0x6b821eb6 +0xda04361b +0xdeed775b +0xf8f07969 +0x2e71b59 +0x801bc45a +0x89ebb84f +0x119ab15c +0x2dbeb297 +0x321463ae +0x4edecb9b +0x5be70da2 +0x6b6b057e +0x71546018 +0x771283ae +0x77a02614 +0x1927f324 +0x36d3e2a6 +0x39e76cf4 +0x69fda74a +0x9cf9c94d +0xa70ef9f3 +0xe3aa64a7 +0xec6485e0 +0xed906881 +0xbad3550 +0x3076b4f3 +0x8c2da750 +0x8c2e4c29 +0x96daa286 +0x97033997 +0xc051a443 +0xc0cb272a +0x1d8a865e +0x7e5b745c +0x555bb46 +0x6d62056 +0xca04c88c +0xab0566 +0x5ec56c8e +0x71b78a48 +0x99015b9d +0xe761f307 +0x29575987 +0x2c6ed036 +0x2ceb17da +0x4af79d3e +0x9a44eb4e +0xafc5a843 +0xbbc8e818 +0xb6b1281 +0x69095277 +0x9554fc0e +0xc6162dd7 +0xd22f87c5 +0xf46ce7b9 +0x56284466 +0x5b2143b4 +0xbe1f8ceb +0xd1de6caf +0x4ecb9b18 +0x6cd6000e +0xafd767f9 +0xfe035f1 +0x13dba3e9 +0x78131334 +0xfcd44e5a +0x3f2719f1 +0x47b59399 +0x6b08be9b +0x95a13403 +0xa94d1b8b +0x2aad457a +0x514378fd +0x62e96e34 +0x74a2d584 +0x795ad09a +0x8fc90bfc +0x9a8a30fc +0x9fa21106 +0xbd767578 +0xc69b1eb4 +0x53f7b0d +0x59d30e5 +0x2d580ff0 +0x47bae63b +0x48f1bd8b +0x728b062f +0xe07144a3 +0xeaea81b9 +0xf5b778ae +0x9a65e07 +0x1203c609 +0x23c2c99e +0x35d51fce +0x80edf66c +0x1897b7e0 +0x8c4d04db +0x8ce7ce8f +0x992f0181 +0xdc3454c5 +0xf39edbd0 +0xb4fd4c6 +0x64ba6dea +0xf27ae161 +0xa93e9aa +0x547a798b +0x8c06c25b +0x9f59e637 +0xacd24172 +0xe7a9b132 +0x149a7566 +0x407d1444 +0x60fa1979 +0x16fd97e0 +0x1cf563cc +0x28e138e0 +0x3dac28ab +0x5aece32b +0x65ad13a9 +0xb87c0d68 +0xc37e7950 +0xed2a85ea +0x33ffa14 +0xb1fd719 +0x529773d3 +0x5a310477 +0x5c910822 +0x64184ea8 +0xf37411ce +0xfd193396 +0xfde83a1b +0x69f43cab +0x729a8c06 +0xb2edf53e +0xb8054629 +0x1e4697bb +0x3a8cf767 +0x54272f9b +0x66048b71 +0xb0078e9b +0xb6f0e885 +0xb7458c39 +0xcab0cdbf +0xefb14707 +0x556bf340 +0x6001d2d7 +0x6a0ddbbb +0x700de653 +0x803ea481 +0xe961b412 +0xda940ef4 +0xdac54173 +0x1eb19fcc +0x4b7f1590 +0x61bddab3 +0x6e547e44 +0x6f814f1a +0x71a6534a +0x7cb95596 +0x84b66eb2 +0xc9e23a5a +0xcfedf9b2 +0xf9f8781a +0x6508f6db +0x7de5502d +0x931bab17 +0xdbfcf165 +0x1efc7261 +0x2935c795 +0x5554ed58 +0x57399dc6 +0x698d2ad9 +0x6c153fff +0x9b03c68a +0xaf2e8605 +0xbff92573 +0xd3c3a6d4 +0x49c193ed +0x80b93cec +0x9aef10a3 +0xabba4af4 +0xb95ff8bb +0xdc00a4d2 +0x29a5e530 +0x71b49971 +0x80846fbb +0xac6dc0d3 +0xbf8b38c8 +0xfc9d39b8 +0xfd89347f +0x41e4222d +0x4b2edd87 +0xa0bc83ac +0xb8a0b5a9 +0xf81f3d71 +0x30173354 +0x4977d429 +0x584d9f5a +0x36e4817 +0x50b99d42 +0x7e74561e +0xe1326f37 +0xfd617acd +0x1baffd3c +0x3677d91d +0x481784c2 +0x50316bb2 +0x845c3f1e +0xb20af5fd +0xd02c2d3f +0xd51ac5c3 +0xf963425c +0xc826a84 +0x5267e08f +0xdfc7e19d +0x48a0d793 +0x76232406 +0x89822328 +0xc6b1ef53 +0xc9cac584 +0xe3346bfd +0xf24992b9 +0x1136bee1 +0x132e76da +0x13d599c2 +0x2c9e0fc4 +0x59e721ba +0x7ed1474e +0x81692e4e +0xa56ee90a +0xae58b399 +0xba05cb86 +0xd0963d44 +0xd688dcc4 +0xeaaaefdc +0xfb6fc3e8 +0xaab91ef +0x2b91f3ca +0x2bca882c +0x580b9ba7 +0xc005f151 +0xe3a15bb5 +0xf31a5c6 +0x81136457 +0x8d9b6abc +0xa8430147 +0xc82e999e +0xa120351 +0x25162677 +0x290f98d2 +0x62d64120 +0x7b048910 +0x7da732e8 +0xa42142b6 +0xbb7b6508 +0xbec6f9a7 +0x261e0e57 +0x399ec435 +0xc36cdec6 +0xe226b1f8 +0xff3efc12 +0xe98f8475 +0xc84a5aa +0xa399ec70 +0xfe3764bf +0x34b462f1 +0x6cecdfbc +0x82c09907 +0x8a914727 +0x8f62260b +0xaaf06dbb +0xe6404c5e +0xf81759c3 +0x844f2870 +0xa31de36c +0x727f3b86 +0x8a373ac1 +0xab29fbea +0x2a78a315 +0x9f462b9c +0xa8ea46c +0x37b85314 +0x138b01b0 +0x92415b9b +0x6f9a01a4 +0xe123b802 +0xe52ad868 +0x203789ab +0x7dbd73df +0x7e5367c +0xdf1a480 +0x72bb210 +0xa0a51906 +0xa65b66eb +0x1b8ca411 +0xe049b330 +0x9dcb784a +0x1cb635e9 +0xc228c5af +0x581cdaf4 +0x7d3d597c +0xc3ca333e +0x946e47f1 +0xf2c128f4 +0x95aba5ec +0x74a673c2 +0x724b594c +0x8d5f9cb5 +0xdd02882c +0xb2b2b8b7 +0xca642b54 +0x6ef835a3 +0x71ecc5a3 +0xcdc76cdf +0xf66ff8ee +0x640944f8 +0xc8b80681 +0xd691df +0x15231688 +0x9d4132bf +0x2ce378ae +0x444d407e +0x9424eb54 +0x6a815b8a +0x71217873 +0x5743ba9c +0xaf769d1b +0xb3ebcdda +0xd9c27be6 +0xedd311c5 +0x69ce1ae3 +0xf9fc681a +0xa834789e +0x2a77cdb3 +0x8d7ed43a +0xba0a4e29 +0x122bba0f +0xf43cc96a +0x5bd32969 +0xb83ec03b +0xef1dc7f2 +0xa39499b9 +0x97066a83 +0x2a5553cb +0x9d14def6 +0x7021fcad +0x845c0ebb +0xc9061a3c +0x495fadd4 +0x70cd03c7 +0x79dbfd34 +0x19b870d6 +0x7242677b +0x9c8d3b5e +0x48cec6da +0xe30fedef +0x626d650e +0xf725884f +0x4f3a8f1d +0xd07096cd +0x67a53ef7 +0xffef37ca +0xf71f85d8 +0xe337bfb3 +0x4a076b16 +0x9d285a65 +0x7e5013ba +0xc3462038 +0x228b6ee4 +0x5efc24cc +0xfd146e16 +0x18b520 +0x91c2a8f1 +0x28627451 +0x4c05d6bf +0x75656739 +0x5a34380b +0xe9741900 +0x78620b41 +0x6fc7632b +0x19e01b15 +0x2a5700eb +0xc97f5b13 +0x6407f465 +0x36b55bc8 +0x6f5fb086 +0xee41e0ed +0xb6cb12f3 +0x721689af +0xbfde4a4f +0xcc9cb32f +0xe87fc210 +0x696076dc +0xe31c29c8 +0x2ad3d771 +0x1c2ecfb8 +0x5ab2e762 +0x20471e78 +0xebf54fca +0xa2cc9f2b +0x4b087292 +0xc8901a9f +0xf7a9cdb5 +0xcdfcb9 +0xb421e164 +0x5870b91 +0xa1085fdc +0xf8da01e4 +0xb48113e6 +0xdbc0122d +0x2354f192 +0x786d45b9 +0xaa475806 +0x190f871e +0x2315b6d5 +0xf37ee1d8 +0xa7f662a4 +0xbad34b5f +0x8822eb05 +0x689959b7 +0xb03b3ab6 +0x259f04dd +0xd3c3dc1b +0xcd71edf2 +0xc277034a +0xa04660d +0xc27fe8e7 +0x17da9c44 +0x35c91877 +0xe15b27a1 +0xb54dee68 +0x39eb80dc +0x2b1d3501 +0x3413e302 +0x1386a486 +0x6484854 +0x4dd308b5 +0x1c13ba29 +0x89b13aa8 +0xa77ee91d +0x4236c51a +0xf2961dc8 +0xd82b9f50 +0xf02e570f +0x3bf1fd7c +0x1221400 +0x84dda883 +0x9907fa91 +0xeab649d2 +0x4444e02b +0xdaad65fc +0x84985f18 +0x3bb0e10b +0x39cd8750 +0x2d00064e +0xcb531ce1 +0x25d3ff63 +0x8d2110 +0x3530770c +0x9179832e +0x3b510d68 +0xdc78512a +0x731331a +0xd48fcef9 +0x38ab4334 +0x662443f6 +0x9263e559 +0xf26531c6 +0x87f48326 +0xec6c409 +0x4dd50367 +0xdbf3ec51 +0x1f3349e1 +0x789f5d09 +0x4f7c8bb5 +0xa37f07de +0xab4ba105 +0x3d67dd41 +0x445f7280 +0x2cb26948 +0x3086a7e5 +0x827cf34e +0x39005781 +0x37cdbf33 +0xf4c28427 +0x85e3d355 +0x35b8f2e5 +0x5d13b978 +0x6fe7e4d +0xab43a647 +0x9e9268ff +0xc8a828fb +0xd6da374f +0x76773e7 +0x69fa1973 +0x5c549280 +0xce72b27 +0xb45c06a9 +0x469d503c +0x6c4ee6cf +0xaccc94cb +0xc688262e +0x3b7c6344 +0x1a820348 +0x9ea45189 +0x8d68be6 +0xc76cdd0 +0x568905b8 +0xa659b9a +0x2a24b75e +0x59e2186c +0xf1925441 +0x84a0a1e7 +0xbeca5133 +0x9eb241d1 +0x52d69cdc +0x2cfe3012 +0x7580848b +0x4feae840 +0xf979c313 +0xf89040ec +0x91ed43e4 +0xac3e0fe2 +0x3d46c408 +0xfbb32e76 +0x78807a32 +0x651ab17e +0x6d09b336 +0xf2fc2d2 +0x57549aa3 +0xc203a723 +0xc6fd9bef +0x6af03075 +0xa160dc98 +0x567fa55d +0x1fec415a +0x7b68c846 +0x1723fe0d +0xaf0c013d +0xce98448b +0x1b7b640f +0x92d6ac37 +0x57582909 +0x400239d8 +0x2bdd1930 +0x91cea369 +0xf40d85f7 +0xd887ff7 +0xcf5f6a25 +0x63ef5730 +0xe04d1c8 +0x90f4397 +0xb3506b41 +0xb0acf09f +0xc677864d +0xc93c065e +0x3c941323 +0x29ea811d +0x3e5b5143 +0x15a5bee4 +0x53da6956 +0x9aa12c81 +0xac340315 +0xab3b5a1e +0xa64d99b9 +0x35d95276 +0x24629247 +0xfc47c589 +0x9a5d40e9 +0xfcf5a2d +0xb8474810 +0x144914dc +0xed279fc9 +0x8ff893a1 +0xab1d5b67 +0xab006cf9 +0xf6adc487 +0xe501077e +0xc083a40d +0x38f29a0d +0x879b51cc +0x2a314da1 +0x8d8169 +0x202abde6 +0x58e282ec +0xbd55a089 +0x8e6f56ac +0xb5088ffc +0x14b49a4c +0x48bed789 +0xf65b5968 +0xff4ba2be +0x37f01fb3 +0x5689a6f8 +0xd9aafc7c +0x1e88426d +0xf7e73df5 +0x1c920449 +0xf4be476e +0x90790e2c +0x3d0f6ca6 +0x4734e26f +0x4efdcbe1 +0x3bfb5a4e +0x5e5390b3 +0x2697ca1b +0x924cbf38 +0xc9ac6f46 +0x174555a5 +0x3b14ef0f +0x9300b35f +0xaa4a6448 +0xb3b076af +0xd2f300a6 +0x9dcea98e +0x7f9eab67 +0x324c191a +0xaa1896b0 +0x7167e30b +0x7b3fd108 +0xb3318639 +0xe83a77a0 +0xe6da62c3 +0x46fbde1c +0x2b290a25 +0x551b6841 +0xa724d2b8 +0x9e111409 +0x77269771 +0x85b0d264 +0x1c1e9975 +0x7367c807 +0x75f36143 +0x5aab2d4 +0xa009aa70 +0xb2297622 +0xaffb7bc7 +0xd5574099 +0x678ec56b +0x603e58ce +0x5e14dfc7 +0xd8007ae1 +0x7310315a +0x6b08fd3c +0xe6e7f6c3 +0x6c7b6bf4 +0x3f5b2651 +0x1702a280 +0x3978dacb +0x6b1e723a +0x87ddda39 +0x1fbdafa8 +0xc5e80876 +0x8675536d +0x7113c47a +0xf7de7989 +0xa13a29c8 +0x72bd11e0 +0x54bd1859 +0x5beecd50 +0x79732465 +0x7b324b4 +0x676a549a +0xc73ff82c +0xfa5ceeb2 +0xeb1dfae4 +0xb5b1530b +0x9d0c03 +0x758daff8 +0x426d7f07 +0x2a0ec857 +0xe7a09ccd +0x974d2152 +0xdb88b2c +0x515bf503 +0xaf0f1606 +0x9f04b44b +0xf3005f6c +0x7d0221b5 +0xb272c0f8 +0x11e5b502 +0xd81a826e +0xb1323805 +0xb6421702 +0x889f1d83 +0xc9245ac6 +0x18ee18e4 +0x460c13a1 +0xe3527c1a +0x6af6efb6 +0x2ed705cd +0xed06a095 +0x8fb7c908 +0xca0bb8a0 +0x268c8411 +0x8a85be47 +0xc9b9aec5 +0x11e0a0f +0x33a3a894 +0x6bf8a6f +0xf4a53b70 +0x96341fb8 +0xbe28207 +0xf3cc75a +0xea073853 +0x335abae2 +0xf25b3813 +0x6c12ba4d +0xb4e2b8dc +0xfcb00932 +0x32a68989 +0x36a72919 +0x81cd264a +0x27755936 +0x24f997f9 +0x5ddc860e +0xe0346663 +0xee38ea04 +0x4db356cc +0x133074af +0x3390dca8 +0x726fcb2f +0x7792336b +0x4ffb1ded +0x7cde9f87 +0x9a2f57b3 +0x92eb9116 +0x7affbece +0x2549058d +0x4fec96ff +0xe118e61e +0xa65f0aca +0xdd9bf693 +0x2b8ac742 +0x967ef100 +0x52f765fa +0x545c6155 + + diff --git a/Ghidra/Features/BSim/data/lshweights_cpool.xml b/Ghidra/Features/BSim/data/lshweights_cpool.xml new file mode 100755 index 0000000000..15b8d51287 --- /dev/null +++ b/Ghidra/Features/BSim/data/lshweights_cpool.xml @@ -0,0 +1,1587 @@ + + + 1.00000000e+00 + 9.99315172e-01 + 9.98629875e-01 + 9.97944107e-01 + 9.97257867e-01 + 9.96571155e-01 + 9.95883970e-01 + 9.95196310e-01 + 9.94508174e-01 + 9.93819562e-01 + 9.93130472e-01 + 9.92440904e-01 + 9.91750857e-01 + 9.91060329e-01 + 9.90369320e-01 + 9.89677828e-01 + 9.88985852e-01 + 9.88293392e-01 + 9.87600447e-01 + 9.86907015e-01 + 9.86213096e-01 + 9.85518688e-01 + 9.84823790e-01 + 9.84128402e-01 + 9.83432521e-01 + 9.82736149e-01 + 9.82039282e-01 + 9.81341920e-01 + 9.80644063e-01 + 9.79945709e-01 + 9.79246856e-01 + 9.78547505e-01 + 9.77847653e-01 + 9.77147300e-01 + 9.76446445e-01 + 9.75745086e-01 + 9.75043223e-01 + 9.74340854e-01 + 9.73637979e-01 + 9.72934595e-01 + 9.72230703e-01 + 9.71526301e-01 + 9.70821388e-01 + 9.70115963e-01 + 9.69410024e-01 + 9.68703571e-01 + 9.67996602e-01 + 9.67289116e-01 + 9.66581113e-01 + 9.65872591e-01 + 9.65163549e-01 + 9.64453985e-01 + 9.63743899e-01 + 9.63033289e-01 + 9.62322155e-01 + 9.61610494e-01 + 9.60898307e-01 + 9.60185591e-01 + 9.59472346e-01 + 9.58758571e-01 + 9.58853246e-01 + 9.58466397e-01 + 9.58338009e-01 + 9.58209899e-01 + 9.58082066e-01 + 9.57954508e-01 + 9.57700217e-01 + 9.57573480e-01 + 9.57447014e-01 + 9.57194890e-01 + 9.56943836e-01 + 9.56818707e-01 + 9.56693842e-01 + 9.56569240e-01 + 9.56320819e-01 + 9.56196999e-01 + 9.56073436e-01 + 9.55950131e-01 + 9.55827082e-01 + 9.55704288e-01 + 9.55459460e-01 + 9.55215639e-01 + 9.55094103e-01 + 9.54972816e-01 + 9.54851776e-01 + 9.54730983e-01 + 9.54610435e-01 + 9.54490132e-01 + 9.54370071e-01 + 9.54250253e-01 + 9.54011340e-01 + 9.53892243e-01 + 9.53773384e-01 + 9.53300313e-01 + 9.53182631e-01 + 9.52481389e-01 + 9.52133843e-01 + 9.52018444e-01 + 9.51903268e-01 + 9.51788315e-01 + 9.51673583e-01 + 9.51330707e-01 + 9.51216852e-01 + 9.51103215e-01 + 9.50989793e-01 + 9.50650818e-01 + 9.50425901e-01 + 9.50313759e-01 + 9.50090106e-01 + 9.49756192e-01 + 9.49534615e-01 + 9.49313857e-01 + 9.49203784e-01 + 9.49093913e-01 + 9.48984244e-01 + 9.48874775e-01 + 9.48656438e-01 + 9.48547568e-01 + 9.48330421e-01 + 9.47898478e-01 + 9.47790977e-01 + 9.47576555e-01 + 9.47469631e-01 + 9.47362898e-01 + 9.47150000e-01 + 9.47043835e-01 + 9.46832067e-01 + 9.46726463e-01 + 9.46515812e-01 + 9.46410764e-01 + 9.46096718e-01 + 9.45784310e-01 + 9.45576938e-01 + 9.45370280e-01 + 9.45267218e-01 + 9.45061622e-01 + 9.44652531e-01 + 9.44550693e-01 + 9.43842615e-01 + 9.43541688e-01 + 9.43341903e-01 + 9.43242259e-01 + 9.43043465e-01 + 9.42746500e-01 + 9.42549334e-01 + 9.42450993e-01 + 9.42352812e-01 + 9.42059225e-01 + 9.41961680e-01 + 9.41767063e-01 + 9.41379703e-01 + 9.41283250e-01 + 9.40994816e-01 + 9.40898977e-01 + 9.40612369e-01 + 9.40517135e-01 + 9.40422051e-01 + 9.40327116e-01 + 9.40137694e-01 + 9.39948864e-01 + 9.39572966e-01 + 9.39385890e-01 + 9.39292569e-01 + 9.39199392e-01 + 9.38920718e-01 + 9.38828112e-01 + 9.38551141e-01 + 9.38367195e-01 + 9.38183807e-01 + 9.38092321e-01 + 9.37727753e-01 + 9.37546288e-01 + 9.37365366e-01 + 9.37005134e-01 + 9.36825818e-01 + 9.36736359e-01 + 9.36379835e-01 + 9.36202356e-01 + 9.36113810e-01 + 9.36025394e-01 + 9.35937107e-01 + 9.35061222e-01 + 9.34800900e-01 + 9.34714374e-01 + 9.34283579e-01 + 9.34197785e-01 + 9.34112112e-01 + 9.33941128e-01 + 9.33855815e-01 + 9.33261953e-01 + 9.33177586e-01 + 9.32841285e-01 + 9.32673827e-01 + 9.32506829e-01 + 9.32423501e-01 + 9.32091325e-01 + 9.32008563e-01 + 9.31925913e-01 + 9.31678633e-01 + 9.31432351e-01 + 9.31350478e-01 + 9.31187060e-01 + 9.31024078e-01 + 9.30537727e-01 + 9.30376467e-01 + 9.30135371e-01 + 9.29895224e-01 + 9.29815384e-01 + 9.29656016e-01 + 9.29497062e-01 + 9.29338520e-01 + 9.29259403e-01 + 9.29101475e-01 + 9.29022664e-01 + 9.28943954e-01 + 9.28708429e-01 + 9.28630122e-01 + 9.28551915e-01 + 9.28473807e-01 + 9.28317891e-01 + 9.28240081e-01 + 9.27775286e-01 + 9.27467370e-01 + 9.27313990e-01 + 9.27160992e-01 + 9.27008376e-01 + 9.26932209e-01 + 9.26856137e-01 + 9.26552790e-01 + 9.26477186e-01 + 9.26401676e-01 + 9.25800913e-01 + 9.25577133e-01 + 9.25502721e-01 + 9.25354165e-01 + 9.24984339e-01 + 9.24690072e-01 + 9.23888017e-01 + 9.23598932e-01 + 9.23096274e-01 + 9.22953408e-01 + 9.22526783e-01 + 9.22173502e-01 + 9.22032754e-01 + 9.21892326e-01 + 9.21752218e-01 + 9.21542648e-01 + 9.20849175e-01 + 9.20642642e-01 + 9.20299945e-01 + 9.20231632e-01 + 9.19959136e-01 + 9.19891199e-01 + 9.19755548e-01 + 9.19485134e-01 + 9.19350369e-01 + 9.19081716e-01 + 9.18747529e-01 + 9.17168037e-01 + 9.17103089e-01 + 9.16908653e-01 + 9.16714823e-01 + 9.16009241e-01 + 9.15691150e-01 + 9.15311576e-01 + 9.14497019e-01 + 9.14372643e-01 + 9.14186543e-01 + 9.13877605e-01 + 9.13754458e-01 + 9.13570191e-01 + 9.13325346e-01 + 9.13203283e-01 + 9.13081458e-01 + 9.13020634e-01 + 9.12717403e-01 + 9.12656933e-01 + 9.12596522e-01 + 9.12536169e-01 + 9.11816444e-01 + 9.11697292e-01 + 9.10987107e-01 + 9.10459708e-01 + 9.10168617e-01 + 9.09994609e-01 + 9.09936714e-01 + 9.09878872e-01 + 9.09821084e-01 + 9.09705667e-01 + 9.09475468e-01 + 9.09418050e-01 + 9.09360684e-01 + 9.08903641e-01 + 9.08393415e-01 + 9.08055554e-01 + 9.07831319e-01 + 9.07775385e-01 + 9.07719501e-01 + 9.07163376e-01 + 9.06174578e-01 + 9.05685944e-01 + 9.05416110e-01 + 9.04986764e-01 + 9.04773183e-01 + 9.04031292e-01 + 9.03611237e-01 + 9.03350115e-01 + 9.03245967e-01 + 9.03038186e-01 + 9.02934551e-01 + 9.01958397e-01 + 9.01805635e-01 + 9.01400062e-01 + 9.01349548e-01 + 9.01299074e-01 + 9.01198246e-01 + 9.01097578e-01 + 9.01047304e-01 + 9.00997070e-01 + 9.00896722e-01 + 9.00746496e-01 + 9.00297943e-01 + 9.00248299e-01 + 8.99704764e-01 + 8.98486495e-01 + 8.98005652e-01 + 8.97433398e-01 + 8.97054734e-01 + 8.96866243e-01 + 8.96725240e-01 + 8.96304093e-01 + 8.95608306e-01 + 8.95562188e-01 + 8.94510589e-01 + 8.94465258e-01 + 8.94239083e-01 + 8.94193944e-01 + 8.94148836e-01 + 8.93923776e-01 + 8.93878859e-01 + 8.93075699e-01 + 8.92633815e-01 + 8.92282491e-01 + 8.91542261e-01 + 8.91196853e-01 + 8.89581073e-01 + 8.89287811e-01 + 8.89037503e-01 + 8.88045893e-01 + 8.87069316e-01 + 8.85948352e-01 + 8.84885539e-01 + 8.83993754e-01 + 8.83878334e-01 + 8.83266213e-01 + 8.83076104e-01 + 8.82622105e-01 + 8.82509100e-01 + 8.82283679e-01 + 8.82246185e-01 + 8.81872429e-01 + 8.81760720e-01 + 8.80837210e-01 + 8.80654072e-01 + 8.79673937e-01 + 8.79458118e-01 + 8.79422218e-01 + 8.79064298e-01 + 8.78213225e-01 + 8.77896913e-01 + 8.77095455e-01 + 8.76818975e-01 + 8.75690695e-01 + 8.75420077e-01 + 8.75251507e-01 + 8.74614868e-01 + 8.74215942e-01 + 8.74149690e-01 + 8.73425296e-01 + 8.72192585e-01 + 8.72000042e-01 + 8.71394014e-01 + 8.70982554e-01 + 8.70919480e-01 + 8.70762059e-01 + 8.70073838e-01 + 8.69794351e-01 + 8.69546907e-01 + 8.68810093e-01 + 8.67930603e-01 + 8.67810207e-01 + 8.67330801e-01 + 8.65942292e-01 + 8.64324977e-01 + 8.63841697e-01 + 8.63700222e-01 + 8.63390031e-01 + 8.62829726e-01 + 8.62025567e-01 + 8.61723061e-01 + 8.60069972e-01 + 8.59218861e-01 + 8.59139622e-01 + 8.58561382e-01 + 8.58535217e-01 + 8.58378439e-01 + 8.58195995e-01 + 8.58169972e-01 + 8.57522658e-01 + 8.56322363e-01 + 8.56019312e-01 + 8.55843161e-01 + 8.55792917e-01 + 8.54919669e-01 + 8.53862101e-01 + 8.52820872e-01 + 8.51866498e-01 + 8.47087582e-01 + 8.46977275e-01 + 8.46669363e-01 + 8.46559732e-01 + 8.45494578e-01 + 8.45193281e-01 + 8.42457246e-01 + 8.42436619e-01 + 8.41861549e-01 + 8.41494394e-01 + 8.40927144e-01 + 8.39866091e-01 + 8.39410685e-01 + 8.38004255e-01 + 8.37811170e-01 + 8.37753350e-01 + 8.36891819e-01 + 8.36625967e-01 + 8.36059708e-01 + 8.35460773e-01 + 8.34922434e-01 + 8.34627203e-01 + 8.34571987e-01 + 8.34461687e-01 + 8.33912809e-01 + 8.33259844e-01 + 8.31706763e-01 + 8.30066835e-01 + 8.29722844e-01 + 8.25366418e-01 + 8.25012660e-01 + 8.24852448e-01 + 8.24389893e-01 + 8.24183525e-01 + 8.22786749e-01 + 8.21661421e-01 + 8.21295341e-01 + 8.20900887e-01 + 8.20674317e-01 + 8.19581935e-01 + 8.19196909e-01 + 8.18858048e-01 + 8.18476931e-01 + 8.17967107e-01 + 8.17202002e-01 + 8.14829863e-01 + 8.14331680e-01 + 8.13631832e-01 + 8.12736224e-01 + 8.11851711e-01 + 8.11400289e-01 + 8.11241641e-01 + 8.10571286e-01 + 8.09648370e-01 + 8.09069514e-01 + 8.04286746e-01 + 8.03737361e-01 + 7.99958876e-01 + 7.99396355e-01 + 7.97865966e-01 + 7.97493793e-01 + 7.95289111e-01 + 7.94174649e-01 + 7.91339747e-01 + 7.90640301e-01 + 7.84947872e-01 + 7.84635414e-01 + 7.83110458e-01 + 7.82430711e-01 + 7.81010829e-01 + 7.79215439e-01 + 7.77487001e-01 + 7.76963705e-01 + 7.76370072e-01 + 7.67857905e-01 + 7.66008509e-01 + 7.54661900e-01 + 7.53708734e-01 + 7.46496333e-01 + 7.41661997e-01 + 7.41593825e-01 + 7.36615306e-01 + 7.35160764e-01 + 7.24037124e-01 + 7.11836060e-01 + 7.05999788e-01 + 6.83462485e-01 + 6.78393662e-01 + 6.74997689e-01 + 6.67759664e-01 + 6.28846493e-01 + 6.23482020e-01 + 5.97169538e-01 + 5.68310973e-01 + 5.48038049e-01 + 1.00000000e+00 + 1.41421356e+00 + 1.60778186e+00 + 1.73205081e+00 + 1.82261573e+00 + 1.89339972e+00 + 1.95124445e+00 + 2.00000000e+00 + 2.04203942e+00 + 2.07892474e+00 + 2.11173664e+00 + 2.14125255e+00 + 2.16804975e+00 + 2.19256811e+00 + 2.21515024e+00 + 2.23606798e+00 + 2.25554048e+00 + 2.27374691e+00 + 2.29083555e+00 + 2.30693045e+00 + 2.32213639e+00 + 2.33654266e+00 + 2.35022594e+00 + 2.36325253e+00 + 2.37568015e+00 + 2.38755936e+00 + 2.39893466e+00 + 2.40984541e+00 + 2.42032663e+00 + 2.43040955e+00 + 2.44012219e+00 + 2.44948974e+00 + 2.45853495e+00 + 2.46727843e+00 + 2.47573888e+00 + 2.48393337e+00 + 2.49187748e+00 + 2.49958547e+00 + 2.50707045e+00 + 2.51434447e+00 + 2.52141865e+00 + 2.52830327e+00 + 2.53500784e+00 + 2.54154119e+00 + 2.54791152e+00 + 2.55412646e+00 + 2.56019313e+00 + 2.56611818e+00 + 2.57190782e+00 + 2.57756788e+00 + 2.58310382e+00 + 2.58852076e+00 + 2.59382352e+00 + 2.59901664e+00 + 2.60410440e+00 + 2.60909082e+00 + 2.61397973e+00 + 2.61877471e+00 + 2.62347919e+00 + 2.62809638e+00 + 2.63262936e+00 + 2.63708102e+00 + 2.64145413e+00 + 2.64575131e+00 + 1.35232660e+01 + 5.06686258e-01 + 5.78081136e+00 + 1.90354965e-01 + 9.33990700e+00 + + +0x713aba7f +0x956dd1b3 +0xae2c48c5 +0xcdd1b007 +0xe2231a80 +0xeac7fe37 +0x56b3354 +0x48cec6da +0x76a56461 +0x82020a30 +0x99041c8c +0xaa16ee49 +0xc55774da +0x885cc42 +0x90f4397 +0xc912980 +0xfd2323f +0x2ecf770f +0x3e24b497 +0x5b91626e +0x5c6c60b9 +0x5fb978cb +0x6a09c71d +0x7db51165 +0x847f6e7e +0xa75eb0d9 +0xa92adf04 +0xaa4a6448 +0xac8b6e21 +0xbd410311 +0xcb44d5eb +0xd1bb124a +0xeb675a07 +0xebe1177 +0x58125dc3 +0x884a90a2 +0xa3a0dc64 +0xb14cdf9b +0xb1d9018a +0xf677da58 +0x2c5807a +0x6b55e76 +0x70fa68d +0x860a849 +0xc1bac29 +0x1db7403c +0x359324ef +0x49e83cb0 +0x5b8b8fd4 +0x64e8e6bf +0x7c402af3 +0x80e7a3d1 +0xc59bb005 +0xd810442a +0xed9e9fe0 +0xf31e4151 +0x16057d8c +0x226c9e2d +0x23dbc342 +0x7799bb41 +0xa10dd8c2 +0xddf95585 +0xe877a232 +0xe8f0070b +0xfa39e55d +0x6c225ee +0xdeefd20 +0x12a18a96 +0x17919484 +0x24f70649 +0x2a16c3fe +0x3c15b1f0 +0x3df0574b +0x589138db +0x5d056722 +0x63ef5730 +0x7c8db460 +0xb30123b6 +0xd176af6b +0xf0eba23e +0x24aef43f +0x404dd39a +0x6ac9db6d +0x6cffb389 +0x886f0433 +0x976eb7aa +0xab97a9a4 +0x117283f7 +0x47b59399 +0x61a949da +0x7319afc5 +0x7561587c +0x78ae8005 +0xd2bfb9a7 +0xdfac0613 +0x184d10fd +0x1e418400 +0x2adde3bc +0x32b56485 +0x38fdbe83 +0x3981bd63 +0x3d772726 +0x50deccc0 +0x5efc24cc +0x6807282d +0x6c7b6bf4 +0x80775189 +0x82cf7c31 +0x8402b269 +0x844fbe10 +0x85f86f59 +0xa6aa05c6 +0xc14106de +0xc395fb15 +0xed67d1e4 +0x1b311961 +0x4abd69d9 +0x6244d7f7 +0xa3aba665 +0xaf6e6351 +0xb54dee68 +0xca505412 +0xeba45add +0xe0a37d1 +0x19248f61 +0x2a797a11 +0x46810e86 +0x9afcc33a +0xb52376c7 +0xc24e1be7 +0xc33f8c1c +0xda6f8ab8 +0xe0d2d656 +0x532d505 +0x953b3138 +0xa34dabf9 +0xb53319f9 +0xecba6461 +0x1ff1a0fe +0x25cb0324 +0x89513213 +0xddb4f722 +0xe19f2a73 +0xf7d4fa93 +0x13645345 +0x1658864d +0x25698076 +0x2aad457a +0xd9b39638 +0xdf89d5e2 +0xec660d7b +0xaa219a6 +0x1defcd66 +0x26170e91 +0x30883ece +0x52e5bdb8 +0x721689af +0x72771929 +0xacf2d738 +0xc2824648 +0xe469c3c8 +0xeff65617 +0x5ac3dcff +0x5aece32b +0x62de31c9 +0xa415e9f6 +0xaa2c0588 +0xaf0c013d +0xb08ec6bd +0xc6e4a522 +0xdf55a96e +0xf65f487e +0x79db410f +0xa8d35f38 +0xe649f1f8 +0x1b4ce03d +0x27431063 +0x5a007c19 +0x6e0e8c2b +0x78526557 +0x8eac1757 +0x927d486c +0x964764c6 +0xae5e68bf +0xd69654a8 +0xe449973d +0xff56cd02 +0x79fff7d +0xd995e6a +0x15a368ac +0x66c566ff +0x7b7da752 +0xb5f71463 +0x14a5604a +0x4459f41d +0x7f384ad8 +0x8f6464e7 +0xae6350cb +0xbc81f715 +0xcd71edf2 +0xe6853ecc +0xed81f90f +0x2cca31fa +0x49137d38 +0xc4f92e21 +0xd3de77cd +0x1c3e0d22 +0x1dcd83c1 +0x2ae64ef5 +0x44e28ac9 +0x72bd11e0 +0x7d6321c8 +0xe1204c15 +0xf45f3422 +0x44d42350 +0x74982243 +0x7adef1ef +0x8165a3dd +0x90ee8455 +0x99df79cd +0xcdbd0f28 +0xd7258d20 +0xe807d9bf +0xee7c91b7 +0xf906d927 +0x1f05c1a7 +0x69888a44 +0x85562089 +0xba21b711 +0x3d245d7f +0x41ec2bf7 +0x5b7eef40 +0x86591d77 +0xaa65338e +0xccd1d17e +0x29ae5d9 +0x1249bd6f +0x64c07a39 +0xa3241d30 +0xb23fb4f3 +0xbc2d03e6 +0xc37e7950 +0xd008e9bc +0xde716599 +0x4174490 +0x3e3c4dae +0xe15b27a1 +0xe8862e42 +0xed2a85ea +0x12c5c413 +0x2a1aaf73 +0x464a240a +0x4efdcbe1 +0x50c62049 +0x544b6e90 +0x63abb87d +0x6747108b +0x71964505 +0xb694c304 +0xbed3758d +0xbf955f94 +0xd133d565 +0x265f1eda +0x7c649c2c +0xbf097751 +0xe761f307 +0xf0f6ffe5 +0x4ab7aee8 +0x8372bc4a +0x92d8eaaf +0xaca230ec +0xd0a92648 +0x711ccc8b +0x77a788e3 +0xa2463519 +0xa9751c69 +0xc92807f9 +0xaa52cdd +0x1688672c +0x1a6a1521 +0x6eee2871 +0x71e10d34 +0xa0e71b1b +0xd582f724 +0xdeece557 +0xe4cd0a9c +0xeb0d22a0 +0x2a2ce354 +0x6a2f4a85 +0x7d16f0e3 +0xb2b2b8b7 +0xbb2762b8 +0xbc238adf +0xe0a14944 +0xb3fd848 +0x2510f9fd +0x4119b6ff +0x5df5c0b2 +0x471f4ba4 +0x89582643 +0xa1b3a460 +0xa50a5aa4 +0xba40054a +0x5b7c4732 +0x621fd94e +0x717f2548 +0x77194b9b +0x7fd99363 +0x83b98616 +0xfb3007fb +0x1c32dc84 +0x263ce460 +0x2cfe3012 +0x339be4de +0x3f025401 +0x72a46f11 +0xabb266bb +0xb2a4d8df +0xbc5055eb +0x28a63aa4 +0x39f4752e +0x460c13a1 +0x7bc85313 +0xcf531808 +0xf12acb26 +0xfef259b3 +0x1ef3b6ca +0x21eb8439 +0x36ad4c95 +0x380faa3e +0x6567d6e2 +0xa4e2fd1e +0xc8266a97 +0x2531a193 +0x33281a09 +0x70bbb76c +0x5fffb5b +0x3fac4162 +0x5bd54600 +0x653af04b +0x834f50e3 +0x8e986285 +0xd252fd31 +0xdbf34d7d +0x51656ae7 +0xca253fea +0xe958d141 +0x120c7a11 +0x3f07af7b +0x51f059fc +0x5e78c184 +0xe1eb6b4a +0xe59c4772 +0xc32e32d +0x1d36ec89 +0xae4621ef +0xd1df8740 +0xed3bb4b8 +0xf6c4ac1b +0x28ab4ad3 +0x53f88dba +0x79cefa94 +0x9fb11f5d +0xb6903729 +0x5142fad3 +0x94ffa17c +0xbfc89ba6 +0xe0f14dec +0xeb5fe524 +0x7beadfa +0x12ac6efa +0x469ef8aa +0x99c566da +0xb9b9cc44 +0xd0d9316 +0x1588852a +0x7c0d3cda +0xaa4cbd10 +0xafa76dc6 +0xc5e50bb4 +0xdb99ee8b +0x3b0363e +0x67d4f964 +0x86063ffe +0x896612b1 +0xb69ef95b +0xbbe7bc52 +0xe62071d0 +0x551e0920 +0x7769b325 +0x8355a9b1 +0x8c571d79 +0x8f6070b4 +0x9a1ff435 +0xe27516b9 +0xec49adfb +0xf1212049 +0x2ceb060f +0x38e3b638 +0x7021fcad +0x81760d95 +0xc1cade03 +0xd40eee1d +0xed37f966 +0x238e7fa +0x173c061f +0x3ce5d4b0 +0x560957d8 +0x5eb87d21 +0x70f9fe49 +0xbb731430 +0xf7a5fcb9 +0x2f3ebbc4 +0x7031d675 +0x7580848b +0x98f9b71e +0xff94bbff +0xceefdf8 +0x5e2081e2 +0x830d1e14 +0xa529d589 +0xb7f132a7 +0x5410b4b9 +0x55ea1d49 +0xb4a7cc73 +0x12bc0ce6 +0x5554ed58 +0x69f43cab +0x7fce49e4 +0xdba7b3b6 +0xf8bf9b13 +0xda410a +0x1a197b45 +0x24df5391 +0x6aa9abf1 +0x7b34b44b +0xfacd4d37 +0x59421149 +0x79bebfc4 +0x8c343615 +0x9b15a11f +0x9e658b78 +0x9f9cc377 +0x21ada994 +0x4f8893d3 +0x569b2b28 +0x63f112f4 +0xa13a29c8 +0xb8dbb442 +0xd8bfe9ba +0x37e349fa +0xedd0f838 +0x27c7fb01 +0x82723a10 +0x71546018 +0xa1203041 +0xd3966674 +0xf3c4e8c +0x2459e5a2 +0xe39e5769 +0x82016328 +0xa8f9b8db +0x1c7a74f8 +0x3ceee46e +0x8279923e +0xab3a0c79 +0xe3e1e364 +0xe5af2cbd +0x310a2bc1 +0xafd818b8 +0xef3df897 +0x6991b617 +0xfb8af9ef +0xebd4d53d +0x2723ca8e +0x30911a1b +0x3360b5dd +0x9cb3df46 +0x617b03d9 +0x8e2b5f1f +0xee2e7f9e +0x217babb3 +0xac5206bc +0x6ee2f53b +0x87b35bd6 +0x863ed66f +0x21ac1de1 +0x5f5bb9b6 +0xd2eb572f +0x2ad3d771 +0xd2150043 +0xcbfcf74 +0x87ddda39 +0xd72e780c +0x4f366202 +0x9fd3deff +0xa89887bc +0x2b4fbb71 +0x99f90715 +0x32da24b6 +0x62ffbfba +0xa4143b32 +0xcdf4102f +0xc739e0c +0x7d039184 +0xdfbadb85 +0x1186833c +0x18f4b2c1 +0x4e0ca2b0 +0xa04dbe0a +0x3076b4f3 +0xd087b456 +0xb2297622 +0xf26531c6 +0xc3ba236c +0xf23b3627 +0x38d5ced5 +0x1fe7ea0e +0x2b86d2cd +0x8742afe9 +0xb2c31190 +0x7a2dc2e2 +0x24f997f9 +0x648b0a6e +0x7f6600c6 +0x743b8d47 +0x1e5b4c55 +0xec168705 +0x513ca6da +0x93a33077 +0xc92e36f7 +0xd0c6c1df +0xecf8fbb6 +0xae0f369 +0x2b6c216e +0x369844a0 +0x9369207f +0x3db56d11 +0x3e3fd83f +0xc1b1658c +0xbb360b7 +0xfa2ec273 +0xe90764d7 +0x4bb14f7 +0x720c1dc6 +0x2effc8e8 +0x8a0a53cf +0x50586692 +0xaa69a80 +0x7242677b +0x7b8c0337 +0x10ec4425 +0xe4b1c9d5 +0xc8dc6e49 +0x22058dab +0x844db7b9 +0xc5a8f3ea +0xc39692b +0xa2466536 +0x48bed789 +0x60e50290 +0x64133388 +0x6398c5bf +0xf127715b +0x237a03ca +0xde186720 +0xe53ebc42 +0x8bacffbf +0x6c4ffee2 +0xfc780674 +0x20940a58 +0xfff86fa1 +0x8a218182 +0x5ed7f89b +0xb1ac6065 +0x5e70d986 +0x66deb51d +0x21e79eb9 +0xb0d3263e +0xa82ecf9c +0x6f85865b +0x6c12af70 +0x2eb2bc7e +0x30b162bd +0x6d0b06b7 +0x88143a18 +0x23e183bf +0x55e75e4a +0x7d546f34 +0x4329261b +0xb87c0d68 +0x758daff8 +0x9179eea7 +0x3d21e870 +0xbc84752b +0x1fcbaa45 +0x50f69be0 +0x9f67c083 +0x9ba33bb4 +0xf3aeb369 +0x1fbdafa8 +0xb272c0f8 +0xd8a8ac25 +0x672f961 +0x675cb69f +0xf172de99 +0x3b0ea898 +0xd8e5f6bf +0xb265fcb3 +0xd93388e5 +0x8929ef2 +0x19b6b9c1 +0x1eda37fb +0x26b20290 +0x65750626 +0x91ade12f +0xb52f27d7 +0x7a11ce4c +0xf7d5df4a +0x2c1ce088 +0xd3922534 +0xf1aec0bb +0x4bea7aa6 +0xae33b71f +0x6b275d22 +0x5e2c839b +0xd9dcedca +0x998e7432 +0x2c7eecc7 +0x1b0f34ca +0x19e01b15 +0x2ed705cd +0x3afdd86f +0x7a7d84fe +0x904dc32c +0xe221404f +0xf24992b9 +0xc677864d +0x88d37721 +0x713c7562 +0x1e22fe70 +0xf49f57b0 +0x974b0cab +0x3dc14c3a +0x839fc74b +0xd27d6f88 +0x2120e398 +0x6b614f0 +0x74a5ba7a +0xf6f76dd1 +0x1221400 +0xc9a49d84 +0xe87bc6c4 +0x15231688 +0xf567bea +0x401a393b +0xc76cdd0 +0x336534b1 +0xe45a31f6 +0x99245b8d +0x80547a21 +0xe2cea64f +0xf8c08d8a +0xb262a112 +0x87c80499 +0xd4f8c136 +0x4778eb38 +0xbde434d8 +0xfa6a0ec5 +0xfff15692 +0xc35312ba +0x41173be9 +0x7d3e5a52 +0x739699b2 +0x52fc2be0 +0xb9017f4f +0x30606f88 +0x1d8bb601 +0x28cd83b6 +0xd86fdd9c +0x4c8fdcc5 +0xc9b44b98 +0x7b0005bb +0xe501077e +0x46b286a2 +0x5afa9a27 +0xc15bc7d2 +0x8d7d6442 +0x9745d90e +0xe4da53d6 +0xf87685e7 +0xfc929e59 +0x6c0e0c6b +0x23ab4c1d +0x59aaa0ff +0xd6f1add1 +0x65d25938 +0xe5843875 +0x39e4a880 +0x9b29efbc +0x9eee927d +0xbeddd7d2 +0xd272ccc2 +0x2d2d5e42 +0xe12d7c8b +0x4896b49 +0xb2c415d1 +0x2c8806d4 +0xa160dc98 +0xfd92d5fb +0x9ab95599 +0xd8aeea9f +0xdc70f080 +0xd793d7c4 +0x7fafdda1 +0x199c3eae +0x1b2b37e +0x7b68c846 +0x52bdb906 +0x81b43ec8 +0xac84408e +0x525b92b7 +0x5d055325 +0xd0963d44 +0x8f3508e0 +0xf997e23d +0x5b685d8c +0xf4dc23f2 +0x79141868 +0xf2fc2d2 +0xef8db9d6 +0xe352d6df +0x534283a5 +0xbe2e1172 +0xac69574d +0xa1576b8e +0xc2685770 +0xcd4bcc63 +0xc0beb6f0 +0x7022d035 +0xc72a388b +0x1cb635e9 +0x6a62bfcb +0x1b24d7ca +0x4df70225 +0x60ef251 +0xe3527c1a +0x16bb6685 +0xf02e570f +0x1702a280 +0xa224cd3a +0x977f4573 +0xfcc4a9ce +0xa6c72899 +0xc26af093 +0xd0a400e7 +0x3ae3d25a +0x7e3425dc +0x6d2f234e +0x4734e26f +0xffbec1c2 +0xc6a94c86 +0x98a68b59 +0x545c6155 +0x4a4795e7 +0xe940d89b +0xa8ad890a +0x94e9f46 +0xf8718885 +0x87ba2cdb +0x44923a5f +0xb94c84ef +0x94309146 +0xf5029acb +0xc7b4bc25 +0x14572bce +0xb4c16d3e +0x39fca908 +0xf0b0cf36 +0x9efaef68 +0x3504f399 +0x175be28a +0xb3318639 +0xaaf292a6 +0x90dfc0a1 +0x5303255d +0x16c62ff1 +0x880ed456 +0x72bce690 +0x6713a186 +0xc350bf62 +0xd1c61fd9 +0x12e3c461 +0xef76cd31 +0xba7cbe7c +0x78dfb936 +0xd700203 +0xe1fc52c0 +0x2705ff15 +0x293d3993 +0xc0f30ae +0x45d32ae9 +0x7b3fd108 +0xfeebb76 +0x44efd1c9 +0x654197cd +0x7ea0cc95 +0x3086a7e5 +0x7def3399 +0x5bfbcc34 +0xb2edf53e +0x70457a3b +0x8b7e7f52 +0x3ee54265 +0xddc7a6e3 +0x12baa537 +0xb5b1530b +0x5e8897d9 +0xc6568a0b +0x87f48326 +0x4b087292 +0x9043e9cb +0xc89a3673 +0xb4baa63d +0x34594da7 +0x2a9a6325 +0x9fc9e71b +0xa45cc8de +0x56752372 +0x4cf7117f +0xd634e966 +0xbb7b6508 +0x54022f68 +0xc032b4cc +0x3b14ef0f +0x774135a4 +0xdcb4724c +0xc4034e3b +0x314d31fc +0xccba12b9 +0x5b21533c +0x1fd1f3c8 +0xd74fb874 +0x5705d804 +0x5c23aea2 +0xb7c69a52 +0x1924ba4c +0x26a99286 +0x568555b5 +0xb97af172 +0x7bdd78c7 +0x96cb2440 +0x93e28195 +0x726fcb2f +0xe7d1f732 +0x4db356cc +0xcb6c4648 +0x6f5fb086 +0x63951de0 +0x3254a3c6 +0x908c36e9 +0xda127b44 +0x1db29d7 +0xd5060fd3 +0xa7ccd883 +0xc83914cd +0x556457c +0xc98c6948 +0x2d472bcd +0xe853ae4e +0x3b7c6344 +0xc87cc6ec +0xaddbf778 +0x582bcf95 +0x75f012f3 +0x9a8a30fc +0x98d27b5a +0xe0346663 +0xeed50b4b +0x87174f94 +0xf1f0e1c +0x44b296e2 +0xd2a4c5b9 +0xc7842d1 +0x66217117 +0x5596b5cd +0xab4ba105 +0xe049d92d +0xdf136708 +0x16d2fede +0x93365cad +0xa64d99b9 +0x5a712a59 +0xb4e2b8dc +0x7da732e8 +0x40498052 +0x485c8fe2 +0xe2c09d07 +0xf6288dee +0x35f05c5c +0x58c751a5 +0x5cfe4007 +0x15a53c6f +0x7cde9f87 +0xa39b5253 +0xf0be154c +0xa6375309 +0x3390dca8 +0xf3005f6c +0x9fa21106 +0x5a1cdf02 +0x5a34380b +0xfcb00932 +0x30008223 +0x69e534b8 +0xcdfcb9 +0x65ad13a9 +0xbd55a089 +0x924b92e4 +0xfb4bc4d3 +0x9ce8e6e6 +0xa6dfc811 +0x2549058d +0xe6e7f6c3 +0x7a2c0fd1 +0xc9c6d713 +0x8fa2ba19 +0x1f3349e1 +0x3a268ab +0xd38a4807 +0x2ec7869b +0x91c2a8f1 +0xea073853 +0x84dda883 +0x3e971b86 +0x8cef517b +0xf71fcf42 +0x94982379 +0x3978dacb +0xc96b9395 +0xeba7f14e +0x6e1fb12c +0x79bdb87 +0x54d29236 +0x2ae36e39 +0x93e58cf7 +0x335abae2 +0x6c12ba4d +0xbcfb1e1e +0x4be143eb +0xc1e5a7a2 +0x4dd308b5 +0xdd9bf693 +0xa65f0aca +0x35318a66 +0xf25b3813 +0x10604963 +0x35fb3383 +0xa7d9833e +0x7af9610d +0x1e675f83 +0xaccc94cb +0xbf1abd18 +0xfe3764bf +0x2b8ac742 +0x3eae4e23 +0xd1f373eb +0x91740b04 +0x405fe769 +0x6af03075 +0x4fec96ff +0x84259f84 +0x30eadb6d +0xe4aab5cb +0x500ec4cd +0xddcc8504 +0xbbd73669 +0x72b0be3e +0xda8bf7f0 +0xf4a416f9 +0x7792336b +0x55a849dc +0xa1120c +0xc0ac89b1 +0xfff16b59 +0x33ffa14 +0xec0a4669 +0x114cedc6 +0xe7e95684 +0x52f765fa +0xbe3c0cf4 +0xc9d6df17 +0xd5574099 +0xd634ab4 +0x7cfa88b4 +0xc9dad914 +0xda04361b +0x19b9ec83 +0xc9c2d512 +0xc9dedb15 + + diff --git a/Ghidra/Features/BSim/data/lshweights_nosize.xml b/Ghidra/Features/BSim/data/lshweights_nosize.xml new file mode 100755 index 0000000000..b04d027eeb --- /dev/null +++ b/Ghidra/Features/BSim/data/lshweights_nosize.xml @@ -0,0 +1,1587 @@ + + + 1.00000000e+00 + 9.99459862e-01 + 9.98919432e-01 + 9.98378710e-01 + 9.97837694e-01 + 9.97296385e-01 + 9.96754783e-01 + 9.96212885e-01 + 9.95670693e-01 + 9.95128205e-01 + 9.94585422e-01 + 9.94042342e-01 + 9.93498965e-01 + 9.92955291e-01 + 9.92411319e-01 + 9.91867048e-01 + 9.91322479e-01 + 9.90777611e-01 + 9.90232442e-01 + 9.89686974e-01 + 9.89141204e-01 + 9.88595133e-01 + 9.88048761e-01 + 9.87502086e-01 + 9.86955108e-01 + 9.86407828e-01 + 9.85860243e-01 + 9.85312354e-01 + 9.84764160e-01 + 9.84215661e-01 + 9.83666856e-01 + 9.83117744e-01 + 9.82568326e-01 + 9.82018600e-01 + 9.81468567e-01 + 9.80918224e-01 + 9.80367574e-01 + 9.79816613e-01 + 9.79265343e-01 + 9.78713762e-01 + 9.78161870e-01 + 9.77609666e-01 + 9.77057151e-01 + 9.76504322e-01 + 9.75951181e-01 + 9.75397726e-01 + 9.74843957e-01 + 9.74289873e-01 + 9.73735473e-01 + 9.73180758e-01 + 9.72625727e-01 + 9.72070378e-01 + 9.71514713e-01 + 9.70958729e-01 + 9.70402426e-01 + 9.69845805e-01 + 9.69288864e-01 + 9.68731602e-01 + 9.68174020e-01 + 9.67616117e-01 + 9.67057891e-01 + 9.66499344e-01 + 9.65940473e-01 + 9.65381279e-01 + 9.64821761e-01 + 9.64261918e-01 + 9.63701749e-01 + 9.63141255e-01 + 9.62580435e-01 + 9.62019288e-01 + 9.62126011e-01 + 9.61973468e-01 + 9.61897344e-01 + 9.61745388e-01 + 9.61518181e-01 + 9.61367192e-01 + 9.61291842e-01 + 9.61141427e-01 + 9.60991393e-01 + 9.60916518e-01 + 9.60841737e-01 + 9.60617958e-01 + 9.60543552e-01 + 9.60469240e-01 + 9.60395020e-01 + 9.60246858e-01 + 9.60172915e-01 + 9.60025305e-01 + 9.59657878e-01 + 9.59074677e-01 + 9.59002177e-01 + 9.58857443e-01 + 9.58713059e-01 + 9.58569026e-01 + 9.58425340e-01 + 9.58281999e-01 + 9.58210459e-01 + 9.58139004e-01 + 9.58067634e-01 + 9.57925152e-01 + 9.57712065e-01 + 9.57641205e-01 + 9.57570430e-01 + 9.57499738e-01 + 9.57288165e-01 + 9.57007230e-01 + 9.56937203e-01 + 9.56867258e-01 + 9.56797395e-01 + 9.56727613e-01 + 9.56657914e-01 + 9.56449301e-01 + 9.56310631e-01 + 9.56172282e-01 + 9.56103227e-01 + 9.56034252e-01 + 9.55965357e-01 + 9.55827805e-01 + 9.55622070e-01 + 9.55280744e-01 + 9.55009080e-01 + 9.54806140e-01 + 9.54738646e-01 + 9.54402315e-01 + 9.54335275e-01 + 9.54268311e-01 + 9.54201421e-01 + 9.54067865e-01 + 9.54001199e-01 + 9.53934606e-01 + 9.53801644e-01 + 9.53536603e-01 + 9.53404521e-01 + 9.53338589e-01 + 9.53272729e-01 + 9.52944514e-01 + 9.52879087e-01 + 9.52683231e-01 + 9.52618088e-01 + 9.52488013e-01 + 9.52358221e-01 + 9.52228708e-01 + 9.52164057e-01 + 9.51906146e-01 + 9.51585309e-01 + 9.51457453e-01 + 9.51393627e-01 + 9.51202556e-01 + 9.51139000e-01 + 9.50948736e-01 + 9.50885448e-01 + 9.50759073e-01 + 9.50695984e-01 + 9.50444292e-01 + 9.50131152e-01 + 9.49944048e-01 + 9.49386204e-01 + 9.49262939e-01 + 9.49201400e-01 + 9.49078512e-01 + 9.48833486e-01 + 9.48772385e-01 + 9.48650369e-01 + 9.48043968e-01 + 9.47983662e-01 + 9.47923417e-01 + 9.47743041e-01 + 9.47683035e-01 + 9.47443609e-01 + 9.47205129e-01 + 9.47086242e-01 + 9.47026886e-01 + 9.46790045e-01 + 9.46436517e-01 + 9.46201977e-01 + 9.46143484e-01 + 9.45619580e-01 + 9.45388184e-01 + 9.45215217e-01 + 9.45157671e-01 + 9.44928033e-01 + 9.44870759e-01 + 9.44528252e-01 + 9.44300987e-01 + 9.44187673e-01 + 9.44074571e-01 + 9.44018100e-01 + 9.43905314e-01 + 9.43568211e-01 + 9.43456259e-01 + 9.43400361e-01 + 9.43232974e-01 + 9.43177281e-01 + 9.43121639e-01 + 9.43066049e-01 + 9.42512926e-01 + 9.42402904e-01 + 9.42347968e-01 + 9.42183458e-01 + 9.42128720e-01 + 9.41964802e-01 + 9.41801326e-01 + 9.41475687e-01 + 9.40883181e-01 + 9.40669148e-01 + 9.40562414e-01 + 9.40509117e-01 + 9.40455867e-01 + 9.40402663e-01 + 9.40296395e-01 + 9.40084415e-01 + 9.39873171e-01 + 9.39820474e-01 + 9.39557673e-01 + 9.39191647e-01 + 9.39035450e-01 + 9.38776008e-01 + 9.38672540e-01 + 9.38311778e-01 + 9.38260415e-01 + 9.38106583e-01 + 9.37698254e-01 + 9.37343193e-01 + 9.37242126e-01 + 9.36990186e-01 + 9.36739283e-01 + 9.36389740e-01 + 9.36339968e-01 + 9.36290236e-01 + 9.35844466e-01 + 9.35795136e-01 + 9.35745845e-01 + 9.35549081e-01 + 9.35499988e-01 + 9.35450935e-01 + 9.35401922e-01 + 9.35304012e-01 + 9.35255115e-01 + 9.34962556e-01 + 9.34768290e-01 + 9.34526321e-01 + 9.34478042e-01 + 9.34045233e-01 + 9.33949467e-01 + 9.33806098e-01 + 9.33758383e-01 + 9.33710705e-01 + 9.33472869e-01 + 9.32999946e-01 + 9.32858777e-01 + 9.32811793e-01 + 9.32764845e-01 + 9.32250770e-01 + 9.32018515e-01 + 9.31787135e-01 + 9.31740964e-01 + 9.31648724e-01 + 9.31143867e-01 + 9.31098176e-01 + 9.30643121e-01 + 9.30462038e-01 + 9.30416850e-01 + 9.30056537e-01 + 9.29966788e-01 + 9.29386581e-01 + 9.29164871e-01 + 9.29032227e-01 + 9.28943956e-01 + 9.27634826e-01 + 9.27548530e-01 + 9.26777243e-01 + 9.26352872e-01 + 9.26183936e-01 + 9.25512766e-01 + 9.25221402e-01 + 9.25138406e-01 + 9.25096950e-01 + 9.24972747e-01 + 9.24890084e-01 + 9.24766295e-01 + 9.24314505e-01 + 9.24191859e-01 + 9.23947293e-01 + 9.23906625e-01 + 9.23420682e-01 + 9.23018615e-01 + 9.22978551e-01 + 9.22658968e-01 + 9.22579328e-01 + 9.22261785e-01 + 9.22024688e-01 + 9.21631525e-01 + 9.21474952e-01 + 9.21396814e-01 + 9.21162985e-01 + 9.20813880e-01 + 9.20620775e-01 + 9.20543700e-01 + 9.20198039e-01 + 9.20121486e-01 + 9.19740120e-01 + 9.19512416e-01 + 9.19436699e-01 + 9.18311829e-01 + 9.17793743e-01 + 9.17756901e-01 + 9.17683282e-01 + 9.17389668e-01 + 9.17170358e-01 + 9.16879137e-01 + 9.15763278e-01 + 9.15514037e-01 + 9.15195034e-01 + 9.15018509e-01 + 9.14807332e-01 + 9.13589432e-01 + 9.13280037e-01 + 9.13143016e-01 + 9.12835805e-01 + 9.12597896e-01 + 9.12563983e-01 + 9.12394688e-01 + 9.12360884e-01 + 9.12091102e-01 + 9.11655156e-01 + 9.11588354e-01 + 9.11521623e-01 + 9.11388373e-01 + 9.10957247e-01 + 9.10792211e-01 + 9.10660492e-01 + 9.10496230e-01 + 9.10168984e-01 + 9.08844731e-01 + 9.08685119e-01 + 9.08621387e-01 + 9.08335385e-01 + 9.08272004e-01 + 9.07641665e-01 + 9.06738678e-01 + 9.05909376e-01 + 9.05271776e-01 + 9.04850240e-01 + 9.04670434e-01 + 9.04104383e-01 + 9.03337861e-01 + 9.03279279e-01 + 9.03103854e-01 + 9.03074664e-01 + 9.02928911e-01 + 9.02870704e-01 + 9.02696400e-01 + 9.02233907e-01 + 9.02032617e-01 + 9.01803346e-01 + 9.01746157e-01 + 9.01574897e-01 + 9.00137448e-01 + 8.99831605e-01 + 8.99665397e-01 + 8.99637738e-01 + 8.99444456e-01 + 8.98977474e-01 + 8.98513875e-01 + 8.98459554e-01 + 8.98161609e-01 + 8.98107585e-01 + 8.97677031e-01 + 8.96798040e-01 + 8.96534039e-01 + 8.96481369e-01 + 8.96402444e-01 + 8.95800559e-01 + 8.95774518e-01 + 8.94536771e-01 + 8.93623838e-01 + 8.92550072e-01 + 8.92130442e-01 + 8.92032100e-01 + 8.91664648e-01 + 8.90911804e-01 + 8.90239269e-01 + 8.90167625e-01 + 8.88891396e-01 + 8.88518031e-01 + 8.88262583e-01 + 8.87113962e-01 + 8.87045687e-01 + 8.86773299e-01 + 8.85739591e-01 + 8.85717301e-01 + 8.84985961e-01 + 8.84481051e-01 + 8.84110354e-01 + 8.82372439e-01 + 8.81950346e-01 + 8.81447404e-01 + 8.81135016e-01 + 8.80782768e-01 + 8.80700161e-01 + 8.80473527e-01 + 8.80432404e-01 + 8.80247673e-01 + 8.80145269e-01 + 8.79940940e-01 + 8.79676259e-01 + 8.79534180e-01 + 8.79291329e-01 + 8.77773780e-01 + 8.77125596e-01 + 8.76871993e-01 + 8.76677576e-01 + 8.75790574e-01 + 8.75351508e-01 + 8.74990992e-01 + 8.73531261e-01 + 8.73106909e-01 + 8.73033386e-01 + 8.72685261e-01 + 8.71705728e-01 + 8.70775957e-01 + 8.70386480e-01 + 8.69876543e-01 + 8.69509706e-01 + 8.68764817e-01 + 8.68575596e-01 + 8.66678967e-01 + 8.66379119e-01 + 8.66130271e-01 + 8.65405618e-01 + 8.64559213e-01 + 8.63691437e-01 + 8.63356981e-01 + 8.62882062e-01 + 8.59874343e-01 + 8.59693572e-01 + 8.57503019e-01 + 8.55933947e-01 + 8.55226181e-01 + 8.53365003e-01 + 8.52832764e-01 + 8.52331641e-01 + 8.52264203e-01 + 8.50665427e-01 + 8.49103272e-01 + 8.49000388e-01 + 8.48195539e-01 + 8.47714612e-01 + 8.47199478e-01 + 8.46291764e-01 + 8.42929045e-01 + 8.42635594e-01 + 8.42343387e-01 + 8.41797365e-01 + 8.40729498e-01 + 8.40718105e-01 + 8.39196987e-01 + 8.39107887e-01 + 8.36916790e-01 + 8.36658444e-01 + 8.35191289e-01 + 8.33097915e-01 + 8.32306082e-01 + 8.32164903e-01 + 8.31983804e-01 + 8.31333708e-01 + 8.31174606e-01 + 8.29428520e-01 + 8.27375762e-01 + 8.27131531e-01 + 8.27056553e-01 + 8.24345938e-01 + 8.23325672e-01 + 8.21926763e-01 + 8.18991284e-01 + 8.17723561e-01 + 8.16437606e-01 + 8.15619545e-01 + 8.14151411e-01 + 8.11159658e-01 + 8.10199258e-01 + 8.08829342e-01 + 8.08611965e-01 + 8.07570414e-01 + 8.06129943e-01 + 8.06039110e-01 + 8.06018165e-01 + 8.04902861e-01 + 8.04655799e-01 + 7.97694900e-01 + 7.97402698e-01 + 7.92511673e-01 + 7.91313536e-01 + 7.85210847e-01 + 7.84373127e-01 + 7.83215812e-01 + 7.81432420e-01 + 7.78664890e-01 + 7.75165441e-01 + 7.67493134e-01 + 7.65810419e-01 + 7.65338625e-01 + 7.60250362e-01 + 7.58902621e-01 + 7.58174495e-01 + 7.55295839e-01 + 7.54614470e-01 + 7.49946498e-01 + 7.46253746e-01 + 7.11502811e-01 + 6.85831755e-01 + 6.69423798e-01 + 1.00000000e+00 + 1.41421356e+00 + 1.60778186e+00 + 1.73205081e+00 + 1.82261573e+00 + 1.89339972e+00 + 1.95124445e+00 + 2.00000000e+00 + 2.04203942e+00 + 2.07892474e+00 + 2.11173664e+00 + 2.14125255e+00 + 2.16804975e+00 + 2.19256811e+00 + 2.21515024e+00 + 2.23606798e+00 + 2.25554048e+00 + 2.27374691e+00 + 2.29083555e+00 + 2.30693045e+00 + 2.32213639e+00 + 2.33654266e+00 + 2.35022594e+00 + 2.36325253e+00 + 2.37568015e+00 + 2.38755936e+00 + 2.39893466e+00 + 2.40984541e+00 + 2.42032663e+00 + 2.43040955e+00 + 2.44012219e+00 + 2.44948974e+00 + 2.45853495e+00 + 2.46727843e+00 + 2.47573888e+00 + 2.48393337e+00 + 2.49187748e+00 + 2.49958547e+00 + 2.50707045e+00 + 2.51434447e+00 + 2.52141865e+00 + 2.52830327e+00 + 2.53500784e+00 + 2.54154119e+00 + 2.54791152e+00 + 2.55412646e+00 + 2.56019313e+00 + 2.56611818e+00 + 2.57190782e+00 + 2.57756788e+00 + 2.58310382e+00 + 2.58852076e+00 + 2.59382352e+00 + 2.59901664e+00 + 2.60410440e+00 + 2.60909082e+00 + 2.61397973e+00 + 2.61877471e+00 + 2.62347919e+00 + 2.62809638e+00 + 2.63262936e+00 + 2.63708102e+00 + 2.64145413e+00 + 2.64575131e+00 + 1.34299587e+01 + 2.67731136e-01 + 6.20184175e-01 + 2.01821663e-02 + 7.10384098e+00 + + +0xd99bb820 +0x26111c79 +0x38ab4334 +0x5e509655 +0x6fc2ae12 +0x71a6534a +0x7c978925 +0x8739361b +0x8f12caba +0xa94d1b8b +0xf2bd8da5 +0x85ff9f1 +0xfcf5a2d +0x1af4eb4e +0x76ca0795 +0x9c9b095f +0xa551f8ad +0xb4da62e7 +0xbd838306 +0xfa3eebd5 +0x2d1a3939 +0x2d5f79f3 +0x39005781 +0x399ec435 +0x46edfdd6 +0x70df4399 +0x71fbcec0 +0x7630eef6 +0x7cc78cda +0x97d94dfb +0x99586fb1 +0x9e81544d +0x9edb5a42 +0xbb58e6c6 +0xbfcef464 +0xc688262e +0xc9d6dc05 +0xcf8ccf4d +0x337ec0f +0x9800896 +0xa521f28 +0x206a9b13 +0x2cba622e +0x30cb7f4c +0x3b510d68 +0x584d9f5a +0x69d31db2 +0x73fe75e7 +0x99958cd8 +0xa31de36c +0xc6b1ef53 +0xf81f3d71 +0x50ea2ed +0x3d382cc3 +0x5acd23ed +0x60758ba6 +0x75df8c17 +0x7af9610d +0xf9f1cb56 +0x27900bd +0x314e3275 +0x40403451 +0x54344f26 +0xae4338b2 +0xcf5c917a +0xdd8846e1 +0xe15ea0d3 +0x1f1f3c5b +0x7b7da752 +0x7f1b1898 +0x8c9db403 +0x8f32ca6f +0xac6dc0d3 +0xb20af5fd +0xd028c78d +0xe3052032 +0x9017338 +0x1710a8dc +0x65d5ba44 +0x78bb038a +0x95f1d4f6 +0x9cf9c94d +0xad66db78 +0xb97a92d2 +0xcbebd059 +0xe4048ee2 +0x15a53c6f +0x2bdd1930 +0x339be4de +0x4cac9c1c +0x518fd0a4 +0x639e9fd5 +0x6e9c26b4 +0x8b7ecdd2 +0x92c1c729 +0x95c3f6c4 +0xbe2d70d7 +0xc07f3ffa +0xc59bb005 +0x3bd14b2 +0x53c6f488 +0x7310315a +0x8e517671 +0x9adaafa8 +0x9c58eda4 +0xeea708a3 +0xfd89541a +0x62d7d1b9 +0x7d457067 +0x958704e5 +0xf8c43d9b +0x1f851ff +0x1cf563cc +0x5bd32969 +0x8c18258a +0xde199f1a +0xf8da01e4 +0x5089 +0x2c071f5e +0x5200ff84 +0x5201b63b +0x6001d2d7 +0x63033208 +0x7de5502d +0x889f1d83 +0x9e9abb04 +0xa0c17cc8 +0xa1a81082 +0xc1eb1afc +0xdc2e70c1 +0xe30fedef +0xfbcd2871 +0x2fd11eef +0x4ef29b06 +0x59e721ba +0x696076dc +0x79f63501 +0x813d1b0e +0x96ebf11b +0xab6485ad +0xacd24172 +0xb83a8fae +0xd83f8e6b +0xefa8f093 +0x138465cb +0x1d4c9fe0 +0x2ee76723 +0x432b245e +0x4bb1a85a +0x523ca5c7 +0x65645ed0 +0x95df6379 +0xd648f188 +0xe3062802 +0xe69e2a0d +0x44f051b +0x2b6d27f9 +0xb37656a6 +0xbfd86b84 +0xe409812b +0xfa90be08 +0x2d1b26fc +0x35a30127 +0x4da69435 +0x57440627 +0x5dbd6fec +0x6d979318 +0x72795795 +0x7ef1efda +0x8d3da149 +0xa0b7a040 +0xb3555ac9 +0xbadbd50c +0xc17b75b0 +0xd3a9e306 +0xdb85e3e8 +0xddb91faf +0xf9800514 +0x18b05696 +0x2c3bffce +0x3764d05b +0x90da81d3 +0xbacff499 +0xbf50a3c5 +0xe4020bd0 +0x931fca +0x205465e1 +0x2ce378ae +0x550c3dd3 +0x69561a5d +0x800fa3e2 +0x95b9eb32 +0xd436e1ab +0xdd828c03 +0xe0fb9e39 +0xeddf10f4 +0xf49f56bd +0x1040cab0 +0x421d3bbf +0x52d10e98 +0x532e0d68 +0x91004568 +0xc48474e2 +0xeb533634 +0x2279d5ee +0x9622bc62 +0xaa475806 +0x4c0c3c6 +0x15231688 +0x1e3600b1 +0x55643e33 +0x6508f6db +0xb6cb12f3 +0xb83ec03b +0xd2555d46 +0xd8e1c945 +0x187fa052 +0x2d6f46db +0x94847e33 +0xdd94f48b +0x7262b86 +0x571045d4 +0x64971440 +0x68b06567 +0xa9948510 +0xcc479751 +0x3d2c810 +0x2338c49f +0x780d0a02 +0x795ad09a +0x95aba5ec +0xb0b584ac +0xdd0d8204 +0xfcd44e5a +0x228be8b +0x25825f76 +0x38ed5bf7 +0x7021fcad +0x7891fd74 +0x96026939 +0xaa5150a4 +0xb30123b6 +0xec6c409 +0x1234f294 +0x71b49971 +0x83dc0dd4 +0xbbc8e818 +0xc8a55c7a +0xcfbea9e8 +0xd749bc20 +0xc0371fa +0x5e2547a9 +0x71c8d20e +0x97d9d39f +0x1baffd3c +0x2a78a315 +0x503c5ec0 +0x57f0d0bf +0x657ba33d +0x879da32c +0xa19ce659 +0xcbe13821 +0xebf54fca +0x1d69f5cb +0x36e51bcd +0x66f977f8 +0x75860f31 +0x7f9eab67 +0x8b68ca30 +0x953b3138 +0x992f0181 +0x9be3ab0c +0xae58b399 +0xe401c070 +0xecb9f4e6 +0xf2385f85 +0xfcf1ce4e +0x5303255d +0xc69b1eb4 +0x626d650e +0x6a0ddbbb +0xb04d0cd8 +0xdba2ceb3 +0x34d03d6e +0x39eb80dc +0x54bca668 +0x699da624 +0x70457a3b +0xef18b9e8 +0xff3fa26d +0x3677d91d +0x4edecb9b +0x7006aec0 +0xd9e7792d +0xdda84672 +0xf11a7333 +0x6b1028d1 +0x8c06c25b +0xc3a02ff9 +0xc7057340 +0xeb6302d6 +0xf4952170 +0x7aeb5823 +0x8167797b +0xbff92573 +0x5f004c1c +0x65c7d387 +0xa1ed21ca +0xc57429bf +0xe52ad868 +0xfee2374e +0x3b5d2033 +0x6cecdfbc +0x9996792d +0xb6b0c0e5 +0xb92e6f65 +0xbe1f8ceb +0xc3dc7db0 +0xe43096c9 +0x2defc373 +0x35d96e87 +0x4e606205 +0x7ed1474e +0x844f2870 +0xa2ef1fbd +0xd8a578d5 +0xfe8da725 +0x922c4f3 +0x23d5b17c +0x29a5e530 +0x2ceb17da +0x98f590d7 +0x9a8a30fc +0x9f117c1e +0x9f462b9c +0xbd502e9b +0xe61bb21c +0xf66ff8ee +0x174e5ef9 +0x46fbde1c +0x7594275e +0x852c2591 +0xd41c1f55 +0xe29dd4b8 +0xea6adde9 +0xab0566 +0x5142fad3 +0x59e2186c +0x70054b29 +0x747f6bdd +0x91d6e493 +0xf40d85f7 +0xf80ac34b +0xfaadcb8d +0xfde83a1b +0xff3efc12 +0x36e4817 +0x4e4cd9ef +0x5be7cd51 +0x91090afc +0x1eda37fb +0x1f05c1a7 +0x1f316e39 +0x966fdb70 +0xc9cac584 +0xcc490fe3 +0xd81a826e +0xe1326f37 +0x184c0559 +0x42e8b450 +0x62b5f4b0 +0x7f93c23c +0x8721785a +0x3d338bfe +0x560edce2 +0x73e37186 +0x81ddb0d7 +0xa7cb153c +0xcb531ce1 +0x2046f71f +0x3bf1fd7c +0x770c0f3a +0x907e5334 +0xdbab4b5f +0xf97d3b10 +0x845c3f1e +0x8822eb05 +0xca642b54 +0x12baa537 +0x45d95945 +0x484c51e3 +0x60b5a14e +0x7c5563df +0x971f55ca +0x357fb6dc +0x5edae4bf +0x7965d209 +0x79f52c05 +0x8ac62901 +0x8f230788 +0x9c53d5dd +0x11e5b502 +0x1e5b4c55 +0x364baa48 +0x3d67dd41 +0x50f69be0 +0x5dcf1ade +0xe326fd59 +0xe97eee9 +0x37b85314 +0x4d69ba48 +0xb82f004f +0xcd71edf2 +0xd691df +0x2d688b7 +0x44ac1b34 +0x6eba4915 +0x7e74561e +0x8a89e43c +0xc9f6095e +0xe638bbe3 +0x6ef835a3 +0x724b594c +0xbb56654a +0xc5d70213 +0xd6bc7e4a +0xf7a9cdb5 +0x2935c795 +0x5c549280 +0x7dd96983 +0xa74d50d6 +0xc36cdec6 +0xdaa2e10f +0x4af9a820 +0x4b2edd87 +0x65a14628 +0x78131334 +0xaaf06dbb +0xf12acb26 +0x2da749b1 +0x2e3cbc4c +0x314d31fc +0x3bfb5a4e +0xb6a19782 +0xbd55a089 +0xe123b802 +0xf5af811d +0x67fd80b +0xf2fc2d2 +0x6089358d +0x6c4ee6cf +0x8d68be6 +0xbad3550 +0xc27a5d6 +0x2ca4522d +0x5228ea90 +0x6ec3f247 +0x81692e4e +0x94985529 +0xf7385b44 +0x579a3017 +0x8a881d71 +0x9bd682e0 +0x9dcb784a +0xf6e96ce8 +0x2fbf7b32 +0x5efc24cc +0xa3de2da2 +0xc170c542 +0xe1bf9484 +0x74356e3 +0x1e09f7e5 +0x75656739 +0xb57b12ac +0xfa6a0ec5 +0x6a815b8a +0x882a5b90 +0xb3772a33 +0x352067b7 +0x6c823adb +0xa6197e7c +0xbfde4a4f +0xe761f307 +0x1fd839a8 +0x23c2c99e +0x261e0e57 +0x43b2095f +0xdc4851f4 +0xf747c40c +0xa3a90ad +0x1163643c +0x1f3349e1 +0x259f04dd +0x9134cf02 +0x9696754d +0xeb1d6c51 +0xa8ea46c +0x786d45b9 +0x9393e8a9 +0xa2cc9f2b +0xb87c0d68 +0xe31c29c8 +0xef7bb384 +0x53444b9f +0x60873d86 +0x9b03c68a +0xb5b520d0 +0xc8901a9f +0xed919554 +0xa93e9aa +0xab91754 +0x1136bee1 +0x51b7c2bb +0x97066a83 +0x7e56eeb1 +0x122bba0f +0xa37f07de +0xd34f1166 +0x202abde6 +0x654197cd +0x43a81a4 +0xe961b412 +0x89634782 +0x92d20d3f +0xb4fd4c6 +0x41e4222d +0xae00d1a2 +0x4e7f6fd7 +0x4a076b16 +0xd3d5e28 +0x2b3417b5 +0xd1354cc0 +0xf11b7e53 +0x5b61e8e8 +0x8543e62e +0xbcbcd1ce +0xe0c7ab61 +0x9d14def6 +0xb0864a3e +0x298a66e7 +0x1b4ce03d +0x155334 +0x727f3b86 +0xf85508b7 +0xf8af29b4 +0xb45c06a9 +0xe049b330 +0x40618840 +0x99f90715 +0x495fadd4 +0x9e9268ff +0xb48113e6 +0x2e71b59 +0xa04660d +0xd9c27be6 +0xd82b9f50 +0xc3521ab +0xbb6e3c78 +0x89ebb84f +0x35d95276 +0x3861fd3b +0x444d407e +0xd8c61975 +0xda04361b +0x88a905e2 +0x2e32c1be +0xc3462038 +0x3aa703d1 +0x1f30b88f +0x13d599c2 +0xa65b66eb +0xa3dafff3 +0x71546018 +0x551b6841 +0xea56cbce +0xf9f8781a +0x24aef43f +0x8a914727 +0xdb20f1f +0x50b99d42 +0x67a53ef7 +0x7243fc91 +0x91ebc792 +0xa56ee90a +0x39cd8750 +0xa62c920c +0xdb6a0b4b +0xca6e3157 +0x731331a +0xdef87cbe +0x9d72eaf9 +0x2660dfff +0x38718bcb +0xe6404c5e +0xec660d7b +0x1c2ecfb8 +0x4dd308b5 +0xeab649d2 +0x9aa12c81 +0x73637d0 +0x91c2a8f1 +0x287915e3 +0x2b4207dc +0x7611449f +0x6340a61f +0x1e4697bb +0x9d285a65 +0xb7458c39 +0x4f1a06ec +0x9079f62d +0xa09bfa67 +0x72910b14 +0x9c8d3b5e +0xa39499b9 +0x44b296e2 +0xf2c128f4 +0xd634ab4 +0x47b59399 +0x25842b60 +0xc35cdd8d +0x6b6b057e +0x8a373ac1 +0x3b14ef0f +0x3bbfbbf +0x78807a32 +0x2b1d3501 +0x630f928 +0x7a6cac2b +0xf1925441 +0x6ee9511b +0xde5852a +0x17da9c44 +0x192244c4 +0xcb0f04a8 +0xaab91ef +0x2b83014f +0x1c13ba29 +0x9fc6f3c7 +0xbe2709dc +0x70e07921 +0x6745d7a8 +0x1b8ca411 +0x78255606 +0xc277034a +0x29a42a00 +0x8c2da750 +0xe226b1f8 +0x4545ab9a +0x301aaec6 +0x19816417 +0x69fa1973 +0xb9017f4f +0x3081ff4a +0xc7f04b79 +0xe7404d61 +0x97b6080a +0xbb7b6508 +0x8c343615 +0x67ac2759 +0x3530770c +0x71217873 +0x890af497 +0xd74c73ba +0x72bb210 +0xab6c5360 +0xc79be60f +0xa8430147 +0x7dbd73df +0x5ab2e762 +0xd1511729 +0xdd6dda1c +0xbad34b5f +0xc9e23a5a +0x5870b91 +0xab43a647 +0xc4e6797 +0x8609fe4e +0x319fca34 +0xa659b9a +0xab4ba105 +0x9907fa91 +0x9424eb54 +0x26a99286 +0x4510aec0 +0x6af03075 +0x2a5700eb +0xf2961dc8 +0x33ffa14 +0x64ba6dea +0xe6e7f6c3 +0x35c91877 +0x138b01b0 +0x2a24b75e +0x5d13b978 +0x2ad3d771 +0x40b7219b +0x7d039184 +0x95a13403 +0x3c4e86 +0x57582909 +0x400239d8 +0x5743ba9c +0xb3ebcdda +0xbc2aeb26 +0x9fa21106 +0xb7383114 +0x4b7f1590 +0x2f21fe1b +0x6b2e56ad +0x25cf4dd7 +0x879b51cc +0x91ed43e4 +0xc4165ad +0x51872616 +0xf39edbd0 +0x3b0363e +0x61a949da +0x84985f18 +0x7e5013ba +0x9e2c4539 +0x2d3768ea +0xdc78512a +0x469d503c +0x190f871e +0x1eb19fcc +0x4444e02b +0xa68fcac0 +0x8c2a4e34 +0xa399ec70 +0x2cb26948 +0x4bcea0fa +0x57cd0486 +0x4be2cf2f +0x568905b8 +0x7b28d399 +0xbec6f9a7 +0xf23cb05f +0x6f9a01a4 +0x2a1dacc9 +0xda439fbf +0x49024bb0 +0xfa08adde +0xcab0cdbf +0x31b192e5 +0x58840a79 +0x721689af +0xce98448b +0x96cb2440 +0xc93c065e +0x172d0010 +0x1acfc672 +0x2acf49f3 +0x651ab17e +0x74a673c2 +0x19e01b15 +0xc203a723 +0xd48fcef9 +0x89b13aa8 +0x7da732e8 +0x983ec133 +0x9d2f0c89 +0x76773e7 +0x149a7566 +0x85e3d355 +0x1c920449 +0x440c37a5 +0x974d2152 +0xac340315 +0x37ac668e +0x79ca8d86 +0xd675c2b5 +0x8ff893a1 +0x52d69cdc +0x977f4573 +0x26c4b02a +0x34b462f1 +0xaf0f1606 +0x6bf8a6f +0xce72b27 +0xc005f151 +0x57399dc6 +0xb54dee68 +0xd887ff7 +0xdaad65fc +0x5118b552 +0x8bc22dc6 +0xdbf3ec51 +0x8d5f9cb5 +0x2b290a25 +0xf26531c6 +0x8a048546 +0x931bab17 +0xedd311c5 +0x567fa55d +0x1efc7261 +0x662443f6 +0x603e58ce +0x5e9bac80 +0xce5181f8 +0x11e0ef8e +0xc0b9d8d9 +0x9263e559 +0x7b68c846 +0x9ea45189 +0xa224cd3a +0x3b0ea898 +0x51622f88 +0xc083a40d +0xaf0c013d +0xaa4a6448 +0xa160dc98 +0xa5eef404 +0x1b7b640f +0x50297ad8 +0x69095277 +0xe87fc210 +0x91cea369 +0x2d00064e +0x54f943ad +0x7367c807 +0xf31a5c6 +0x3c941323 +0xd0611cf2 +0x803ea481 +0x7580848b +0x789f5d09 +0xb0acf09f +0x9dcea98e +0x25162677 +0x4dd50367 +0xbd767578 +0x87f48326 +0x3f5b2651 +0xab006cf9 +0xf6adc487 +0xab3b5a1e +0x548ec995 +0xa1120c +0x1fbdafa8 +0x7242677b +0x1cb635e9 +0xf02e570f +0x2cfe3012 +0xf4c28427 +0x9179832e +0x3086a7e5 +0x6c153fff +0x49c193ed +0xdf04a43d +0xc5e80876 +0x3dde6cc8 +0x827cf34e +0xf4be476e +0xc76cdd0 +0x2697ca1b +0x8d2110 +0xe7a09ccd +0xdac54173 +0x5e70d986 +0x2354f192 +0xfbb32e76 +0x58e282ec +0xff4ba2be +0x6c7b6bf4 +0xe501077e +0x65ad13a9 +0xc8a828fb +0x5beecd50 +0xa92829a2 +0xdf1a480 +0x6b1e723a +0xe15b27a1 +0x97033997 +0x324c191a +0x5efa66a8 +0x5e5390b3 +0x57549aa3 +0xc385c6bc +0xf7de7989 +0x14b49a4c +0x8bc13632 +0x582bcf95 +0x1723fe0d +0x72bd11e0 +0xcc644f7f +0x841a0b8b +0x16b08754 +0x7167e30b +0x758daff8 +0xe87a72e5 +0x45993b76 +0x7b3fd108 +0x1702a280 +0x1c1e9975 +0xcdfcb9 +0x200f0c98 +0xb03b3ab6 +0x6b08fd3c +0x2a0ec857 +0x5aab2d4 +0x3b7c6344 +0x4734e26f +0x6617f864 +0x3978dacb +0xaffb7bc7 +0xb8474810 +0xbeca5133 +0x90f4397 +0x48bed789 +0x460c13a1 +0x5a34380b +0x4a1351b9 +0xba0a4e29 +0xf43cc96a +0x8e6f56ac +0x8675536d +0xb3b076af +0x6af6efb6 +0x84dda883 +0xc9245ac6 +0xb3318639 +0x63ef5730 +0x676a549a +0x4efdcbe1 +0x87ddda39 +0x9f04b44b +0xc9b9aec5 +0x7c0182d0 +0x18ee18e4 +0x8d8169 +0x4236c51a +0xe118e61e +0xb5b1530b +0x515bf503 +0x8fb7c908 +0xb6421702 +0x96341fb8 +0x54bd1859 +0xb2297622 +0xb272c0f8 +0x8a85be47 +0xed06a095 +0x7cb95596 +0x18b520 +0x700de653 +0x6f5fb086 +0xa13a29c8 +0x3b2e1e6d +0xaccc94cb +0xf4a53b70 +0xa009aa70 +0xf25b3813 +0xb1323805 +0x24f997f9 +0xea073853 +0x7b324b4 +0xc6fd9bef +0x2ed705cd +0xf3005f6c +0x33a3a894 +0xb4e2b8dc +0x3bb0e10b +0x38f29a0d +0x5ddc860e +0xe3527c1a +0xf3cc75a +0x3390dca8 +0x2a314da1 +0xd2f300a6 +0xfcb00932 +0xca0bb8a0 +0xee38ea04 +0x9d0c03 +0xd5574099 +0x81cd264a +0x32a68989 +0x268c8411 +0x335abae2 +0x6c12ba4d +0x4db356cc +0x7affbece +0xbe28207 +0x133074af +0x7792336b +0x726fcb2f +0xe0346663 +0x90790e2c +0x4fec96ff +0x92eb9116 +0x7cde9f87 +0xa65f0aca +0x2549058d +0xdd9bf693 +0x9a2f57b3 +0x36a72919 +0x2b8ac742 +0x4ffb1ded +0x52f765fa +0x967ef100 +0x545c6155 + + diff --git a/Ghidra/Features/BSim/data/medium_32.xml b/Ghidra/Features/BSim/data/medium_32.xml new file mode 100755 index 0000000000..bfc9a0b72b --- /dev/null +++ b/Ghidra/Features/BSim/data/medium_32.xml @@ -0,0 +1,13 @@ + + + Medium 32-bit + Example Owner + A medium sized (~10 million functions) database tuned for 32-bit executables + 0 + 0 + 0x49 + +17 +146 +lshweights_32.xml + diff --git a/Ghidra/Features/BSim/data/medium_64.xml b/Ghidra/Features/BSim/data/medium_64.xml new file mode 100755 index 0000000000..217b44c8ee --- /dev/null +++ b/Ghidra/Features/BSim/data/medium_64.xml @@ -0,0 +1,13 @@ + + + Medium 64-bit + Example Owner + A medium sized (~10 million functions) database tuned for 64-bit executables + 0 + 0 + 0x49 + +17 +146 +lshweights_64.xml + diff --git a/Ghidra/Features/BSim/data/medium_cpool.xml b/Ghidra/Features/BSim/data/medium_cpool.xml new file mode 100755 index 0000000000..9118a961ef --- /dev/null +++ b/Ghidra/Features/BSim/data/medium_cpool.xml @@ -0,0 +1,13 @@ + + + Medium JVM/Dalvik + Example Owner + A medium sized (~10 million functions) database tuned for java .class or .dex files + 0 + 0 + 0x49 + +17 +146 +lshweights_cpool.xml + diff --git a/Ghidra/Features/BSim/data/medium_nosize.xml b/Ghidra/Features/BSim/data/medium_nosize.xml new file mode 100755 index 0000000000..f7f2d48f98 --- /dev/null +++ b/Ghidra/Features/BSim/data/medium_nosize.xml @@ -0,0 +1,13 @@ + + + Medium No Size + Example Owner + A medium sized (~10 million functions) database tuned for executables with different address/register sizes + 0 + 0 + 0x4d + +17 +146 +lshweights_nosize.xml + diff --git a/Ghidra/Features/BSim/data/serverconfig.xml b/Ghidra/Features/BSim/data/serverconfig.xml new file mode 100755 index 0000000000..818e7371de --- /dev/null +++ b/Ghidra/Features/BSim/data/serverconfig.xml @@ -0,0 +1,14 @@ + + 2GB + 16MB + 30min + '*' + on + + scram-sha-256 + + + + + + diff --git a/Ghidra/Features/BSim/ghidra_scripts/AddProgramToH2BSimDatabaseScript.java b/Ghidra/Features/BSim/ghidra_scripts/AddProgramToH2BSimDatabaseScript.java new file mode 100644 index 0000000000..f031634822 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/AddProgramToH2BSimDatabaseScript.java @@ -0,0 +1,175 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +//Generate BSim signatures for the current program. The URL for the program is +//created from the local storage location. These signatures are intended for the +//in-memory database backend. +//@category BSim +import java.io.File; +import java.io.IOException; +import java.net.URL; +import java.util.Iterator; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.script.GhidraScript; +import ghidra.features.base.values.GhidraValuesMap; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.FunctionDatabase.Error; +import ghidra.features.bsim.query.FunctionDatabase.ErrorCategory; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager.BSimH2FileDataSource; +import ghidra.features.bsim.query.protocol.*; +import ghidra.framework.model.DomainFolder; +import ghidra.framework.protocol.ghidra.GhidraURL; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.FunctionManager; +import ghidra.util.MessageType; +import ghidra.util.Msg; + +//@category BSim +//Generates and commits the BSim signatures for the currentProgram to the +//selected H2 BSim database +public class AddProgramToH2BSimDatabaseScript extends GhidraScript { + + private static final String DATABASE = "H2 Database"; + + @Override + protected void run() throws Exception { + if (isRunningHeadless()) { + popup("Use the \"bsim\" command-line tool to add programs to a database headlessly"); + return; + } + + if (currentProgram == null) { + popup("This script requires that a program be open in the tool"); + return; + } + + GhidraValuesMap values = new GhidraValuesMap(); + values.defineFile(DATABASE, null, new File(System.getProperty("user.home"))); + values.setValidator((valueMap, status) -> { + File selected = valueMap.getFile(DATABASE); + if (selected.isDirectory() || + !selected.getAbsolutePath().endsWith(BSimServerInfo.H2_FILE_EXTENSION)) { + status.setStatusText("Invalid Database File!", MessageType.ERROR); + return false; + } + return true; + }); + + askValues("Select Database File", null, values); + + File h2DbFile = values.getFile(DATABASE); + + FunctionDatabase h2Database = null; + try { + BSimServerInfo serverInfo = + new BSimServerInfo(DBType.file, null, 0, h2DbFile.getAbsolutePath()); + h2Database = BSimClientFactory.buildClient(serverInfo, false); + BSimH2FileDataSource bds = + BSimH2FileDBConnectionManager.getDataSourceIfExists(h2Database.getServerInfo()); + if (bds == null) { + popup(h2DbFile.getAbsolutePath() + " is not an H2 database file"); + return; + } + if (bds.getActiveConnections() > 0) { + popup("There is an existing connection to the database."); + return; + } + + h2Database.initialize(); + DatabaseInformation dbInfo = h2Database.getInfo(); + + LSHVectorFactory vectorFactory = h2Database.getLSHVectorFactory(); + GenSignatures gensig = null; + try { + gensig = new GenSignatures(dbInfo.trackcallgraph); + gensig.setVectorFactory(vectorFactory); + gensig.addExecutableCategories(dbInfo.execats); + gensig.addFunctionTags(dbInfo.functionTags); + gensig.addDateColumnName(dbInfo.dateColumnName); + + DomainFolder df = currentProgram.getDomainFile().getParent(); + URL folderURL = df.getSharedProjectURL(); + if (folderURL == null) { + folderURL = df.getLocalProjectURL(); + } + String path = GhidraURL.getProjectPathname(folderURL); + + URL normalizedProjectURL = GhidraURL.getProjectURL(folderURL); + String repo = normalizedProjectURL.toExternalForm(); + + gensig.openProgram(this.currentProgram, null, null, null, repo, path); + final FunctionManager fman = currentProgram.getFunctionManager(); + final Iterator iter = fman.getFunctions(true); + gensig.scanFunctions(iter, fman.getFunctionCount(), monitor); + final DescriptionManager manager = gensig.getDescriptionManager(); + + //need to call sortCallGraph on each FunctionDescription + //this de-dupes the list of callees for each function + //without this there can be SQL errors due to inserting duplicate + //entries into the callgraph table + manager.listAllFunctions().forEachRemaining(fd -> fd.sortCallgraph()); + + InsertRequest insertreq = new InsertRequest(); + insertreq.manage = manager; + if (insertreq.execute(h2Database) == null) { + Error lastError = h2Database.getLastError(); + if ((lastError.category == ErrorCategory.Format) || + (lastError.category == ErrorCategory.Nonfatal)) { + Msg.showWarn(this, null, "Skipping Insert", + currentProgram.getName() + ": " + lastError.message); + return; + } + throw new IOException(currentProgram.getName() + ": " + lastError.message); + } + + StringBuffer status = new StringBuffer(currentProgram.getName()); + status.append(" added to database "); + status.append(dbInfo.databasename); + status.append("\n\n"); + QueryExeCount exeCount = new QueryExeCount(); + ResponseExe countResponse = exeCount.execute(h2Database); + if (countResponse != null) { + status.append(dbInfo.databasename); + status.append(" contains "); + status.append(countResponse.recordCount); + status.append(" executables."); + } + else { + status.append("null response from QueryExeCount"); + } + popup(status.toString()); + } + finally { + if (gensig != null) { + gensig.dispose(); + } + } + + } + finally { + if (h2Database != null) { + h2Database.close(); + BSimH2FileDataSource bds = + BSimH2FileDBConnectionManager.getDataSourceIfExists(h2Database.getServerInfo()); + bds.dispose(); + } + } + } +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/CompareExecutables.java b/Ghidra/Features/BSim/ghidra_scripts/CompareExecutables.java new file mode 100755 index 0000000000..cd88d84b0e --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/CompareExecutables.java @@ -0,0 +1,80 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Calculate similarity/signifigance scores between executables by +// combining their function scores. +//@category BSim + +import java.net.URL; + +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.query.BSimClientFactory; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.client.*; +import ghidra.features.bsim.query.description.ExecutableRecord; + +public class CompareExecutables extends GhidraScript { + + private ExecutableComparison exeCompare; + @Override + protected void run() throws Exception { + URL url = BSimClientFactory.deriveBSimURL("ghidra://localhost/repo"); + try (FunctionDatabase database = BSimClientFactory.buildClient(url, true)) { + // FileScoreCaching cache = new FileScoreCaching("/tmp/test_scorecacher.txt"); + TableScoreCaching cache = new TableScoreCaching(database); + exeCompare = + new ExecutableComparison(database, 1000000, "11111111111111111111111111111111", + cache, + monitor); + // Specify the list of executables to compare by giving their md5 hash + // exeCompare.addExecutable("22222222222222222222222222222222"); // 32 hex-digit string + // exeCompare.addExecutable("33333333333333333333333333333333"); + exeCompare.addAllExecutables(5000); + ExecutableScorer scorer = exeCompare.getScorer(); + if (!exeCompare.isConfigured()) { + exeCompare.resetThresholds(0.7, 10.0); + } + exeCompare.fillinSelfScores(); // Prefetch self-scores, calculate any we are missing + + exeCompare.performScoring(); + scorer.commitSelfScore(); // Commit the newly calculated self-score + + println("Maximum cluster size = " + Integer.toString(exeCompare.getMaxHitCount())); + println("Hit count exceeded = " + Integer.toString(exeCompare.getExceedCount())); + float scoreThresh = 0.01f; + int numExe = scorer.numExecutables(); + ExecutableRecord exeA = scorer.getSingularExecutable(); + float selfScoreA = scorer.getSingularSelfScore(); + for (int i = 1; i <= numExe; ++i) { + ExecutableRecord exeB = scorer.getExecutable(i); + float selfScoreB = scorer.getScore(i); + if (selfScoreB == 0.0f) { // This is possible if the executable has no "rare" functions. + continue; // as defined by the ExecutableComparison.hitCountThreshold + } + ExecutableRecord smallRecord = selfScoreA < selfScoreB ? exeA : exeB; + ExecutableRecord bigRecord = selfScoreA < selfScoreB ? exeB : exeA; + float libScore = scorer.getNormalizedScore(i, true); + float totalScore = scorer.getNormalizedScore(i, false); + if (libScore < scoreThresh) { + continue; + } + println(smallRecord.getNameExec() + " " + bigRecord.getNameExec()); + println(" " + Float.toString(libScore) + " library score"); + println(" " + Float.toString(totalScore) + " total score"); + } + } + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/CompareSignatures.java b/Ghidra/Features/BSim/ghidra_scripts/CompareSignatures.java new file mode 100755 index 0000000000..21dc52125d --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/CompareSignatures.java @@ -0,0 +1,148 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Use the decompiler to generate a signature for the current function containing the cursor +// If we remember the last signature that was generated, compare this signature with +// the last signature and print the similarity +//@category BSim + +import java.io.*; + +import org.xml.sax.SAXException; + +import generic.jar.ResourceFile; +import generic.lsh.vector.*; +import ghidra.app.decompiler.DecompInterface; +import ghidra.app.decompiler.DecompileOptions; +import ghidra.app.decompiler.signature.SignatureResult; +import ghidra.app.script.GhidraScript; +import ghidra.app.services.ProgramManager; +import ghidra.features.bsim.query.GenSignatures; +import ghidra.program.model.address.Address; +import ghidra.program.model.lang.LanguageID; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +public class CompareSignatures extends GhidraScript { + + private LSHVectorFactory vectorFactory; + + private LSHVector generateVector(Function f, Program program) { + DecompInterface decompiler = new DecompInterface(); + decompiler.setOptions(new DecompileOptions()); + decompiler.toggleSyntaxTree(false); + decompiler.setSignatureSettings(vectorFactory.getSettings()); + if (!decompiler.openProgram(program)) { + println("Unable to initalize the Decompiler interface"); + println(decompiler.getLastMessage()); + return null; + } + SignatureResult sigres = decompiler.generateSignatures(f, false, 10, null); + LSHVector vec = vectorFactory.buildVector(sigres.features); + return vec; + } + + private Program getProgram(Program[] progarray, String name) { + if ((name == null) || (progarray == null)) { + return null; + } + for (Program prog : progarray) { + if (name.equals(prog.getName())) { + return prog; + } + } + return null; + } + + private static void readWeights(LSHVectorFactory vectorFactory, ResourceFile weightsFile) + throws FileNotFoundException, IOException, SAXException { + InputStream input = weightsFile.getInputStream(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(input, "Vector weights parser", + SpecXmlUtils.getXmlHandler(), false); + vectorFactory.readWeights(parser); + input.close(); + } + + private void buildLSHVectorFactory() { + vectorFactory = new WeightedLSHCosineVectorFactory(); + try { + LanguageID id = currentProgram.getLanguageID(); + ResourceFile defaultWeightsFile = GenSignatures.getWeightsFile(id, id); + readWeights(vectorFactory, defaultWeightsFile); + } + catch (FileNotFoundException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + catch (IOException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + catch (SAXException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + } + + @Override + protected void run() throws Exception { + Function func = this.getFunctionContaining(this.currentAddress); + if (func == null) { + return; + } + buildLSHVectorFactory(); + LSHVector vec = generateVector(func, currentProgram); + ProgramManager programManager = state.getTool().getService(ProgramManager.class); + Program[] progarray = programManager.getAllOpenPrograms(); + String lastprogram_string = System.getProperty("ghidra.lastprogram"); + Program lastprogram = getProgram(progarray, lastprogram_string); + VectorCompare veccompare = new VectorCompare(); + if (lastprogram != null) { + String addrstring = System.getProperty("ghidra.lastaddress"); + if (addrstring != null) { + Address addr = lastprogram.getAddressFactory().getAddress(addrstring); + Function lastfunction = lastprogram.getFunctionManager().getFunctionAt(addr); + if (lastfunction != null) { + LSHVector lastvector = generateVector(lastfunction, lastprogram); + double sim = lastvector.compare(vec, veccompare); + double signif = vectorFactory.calculateSignificance(veccompare); + StringBuilder buf = new StringBuilder(); + buf.append("Comparison results:\n"); + buf.append(lastprogram.getName()); + buf.append("."); + buf.append(lastfunction.getName()); + buf.append(" vs. "); + buf.append(currentProgram.getName()); + buf.append("."); + buf.append(func.getName()); + buf.append("\n Similarity: "); + buf.append(Double.toString(sim)); + buf.append("\n Significance: "); + buf.append(Double.toString(signif)); + buf.append("\n"); + lastvector.compareDetail(vec, buf); + println(buf.toString()); + } + } + } + System.setProperty("ghidra.lastprogram", currentProgram.getName()); + String addrstring = func.getEntryPoint().toString(); + System.setProperty("ghidra.lastaddress", addrstring); + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/CompareSignaturesSpecifyWeights.java b/Ghidra/Features/BSim/ghidra_scripts/CompareSignaturesSpecifyWeights.java new file mode 100755 index 0000000000..3886b2133e --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/CompareSignaturesSpecifyWeights.java @@ -0,0 +1,155 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Compare the BSim feature vectors of two functions. +//@category BSim + +import java.io.*; + +import org.xml.sax.SAXException; + +import generic.jar.ResourceFile; +import generic.lsh.vector.*; +import ghidra.app.decompiler.DecompInterface; +import ghidra.app.decompiler.DecompileOptions; +import ghidra.app.decompiler.signature.SignatureResult; +import ghidra.app.script.GhidraScript; +import ghidra.app.services.ProgramManager; +import ghidra.framework.Application; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.util.exception.CancelledException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +public class CompareSignaturesSpecifyWeights extends GhidraScript { + + private static final String DEFAULT_LSH_WEIGHTS_FILE = "lshweights_nosize.xml"; + private LSHVectorFactory vectorFactory; + + private LSHVector generateVector(Function f, Program program) { + DecompInterface decompiler = new DecompInterface(); + decompiler.setOptions(new DecompileOptions()); + decompiler.setSignatureSettings(vectorFactory.getSettings()); + decompiler.toggleSyntaxTree(false); + if (!decompiler.openProgram(program)) { + println("Unable to initalize the Decompiler interface"); + println(decompiler.getLastMessage()); + return null; + } + + SignatureResult sigres = decompiler.generateSignatures(f, false, 10, null); + + LSHVector vec = vectorFactory.buildVector(sigres.features); + return vec; + } + + private static void readWeights(LSHVectorFactory vectorFactory, ResourceFile weightsFile) + throws FileNotFoundException, IOException, SAXException { + InputStream input = weightsFile.getInputStream(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(input, "Vector weights parser", + SpecXmlUtils.getXmlHandler(), false); + vectorFactory.readWeights(parser); + input.close(); + } + + private boolean buildLSHVectorFactory() { + vectorFactory = new WeightedLSHCosineVectorFactory(); + try { + String weightsFile = + askString("Enter weights file name", "weights file", DEFAULT_LSH_WEIGHTS_FILE); + ResourceFile defaultWeightsFile = Application.findDataFileInAnyModule(weightsFile); + readWeights(vectorFactory, defaultWeightsFile); + } + catch (FileNotFoundException e) { + e.printStackTrace(); + return false; + } + catch (IOException e) { + e.printStackTrace(); + return false; + } + catch (SAXException e) { + e.printStackTrace(); + return false; + } + catch (CancelledException e) { + return false; + } + return true; + } + + private Program getProgram(Program[] progarray, String name) { + if ((name == null) || (progarray == null)) { + return null; + } + for (Program prog : progarray) { + if (name.equals(prog.getName())) { + return prog; + } + } + return null; + } + + @Override + protected void run() throws Exception { + Function func = this.getFunctionContaining(this.currentAddress); + if (func == null) { + return; + } + if (!buildLSHVectorFactory()) { + return; + } + LSHVector vec = generateVector(func, currentProgram); + ProgramManager programManager = state.getTool().getService(ProgramManager.class); + Program[] progarray = programManager.getAllOpenPrograms(); + String lastprogram_string = System.getProperty("ghidra.lastprogram"); + Program lastprogram = getProgram(progarray, lastprogram_string); + VectorCompare veccompare = new VectorCompare(); + if (lastprogram != null) { + String addrstring = System.getProperty("ghidra.lastaddress"); + if (addrstring != null) { + Address addr = lastprogram.getAddressFactory().getAddress(addrstring); + Function lastfunction = lastprogram.getFunctionManager().getFunctionAt(addr); + if (lastfunction != null) { + LSHVector lastvector = generateVector(lastfunction, lastprogram); + double sim = lastvector.compare(vec, veccompare); + double signif = vectorFactory.calculateSignificance(veccompare); + StringBuilder buf = new StringBuilder(); + buf.append("Comparison results:\n"); + buf.append(lastprogram.getName()); + buf.append("."); + buf.append(lastfunction.getName()); + buf.append(" vs. "); + buf.append(currentProgram.getName()); + buf.append("."); + buf.append(func.getName()); + buf.append("\n Similarity: "); + buf.append(Double.toString(sim)); + buf.append("\n Significance: "); + buf.append(Double.toString(signif)); + buf.append("\n"); + lastvector.compareDetail(vec, buf); + println(buf.toString()); + } + } + } + System.setProperty("ghidra.lastprogram", currentProgram.getName()); + String addrstring = func.getEntryPoint().toString(); + System.setProperty("ghidra.lastaddress", addrstring); + } +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/CreateH2BSimDatabaseScript.java b/Ghidra/Features/BSim/ghidra_scripts/CreateH2BSimDatabaseScript.java new file mode 100644 index 0000000000..ef332f6d64 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/CreateH2BSimDatabaseScript.java @@ -0,0 +1,170 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +//Creates an empty file-based H2 BSim database +//@category BSim +import java.io.File; +import java.io.IOException; +import java.util.*; + +import org.apache.commons.lang3.StringUtils; + +import ghidra.app.script.GhidraScript; +import ghidra.features.base.values.GhidraValuesMap; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.FunctionDatabase.Error; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager.BSimH2FileDataSource; +import ghidra.features.bsim.query.protocol.*; +import ghidra.util.MessageType; +import ghidra.util.Msg; + +public class CreateH2BSimDatabaseScript extends GhidraScript { + private static final String NAME = "Database Name"; + private static final String DIRECTORY = "Database Directory"; + private static final String DATABASE_TEMPLATE = "Database Template"; + private static final String FUNCTION_TAGS = "Function Tags (CSV)"; + private static final String EXECUTABLE_CATEGORIES = "Executable Categories (CSV)"; + + private static final String[] templates = + { "medium_nosize", "medium_32", "medium_64", "medium_cpool" }; + + @Override + protected void run() throws Exception { + if (isRunningHeadless()) { + popup("Use \"bsim\" to create an H2 BSim database from the command line"); + return; + } + + GhidraValuesMap values = new GhidraValuesMap(); + values.defineString(NAME, ""); + values.defineDirectory(DIRECTORY, new File(System.getProperty("user.home"))); + values.defineChoice(DATABASE_TEMPLATE, "medium_nosize", templates); + values.defineString(FUNCTION_TAGS); + values.defineString(EXECUTABLE_CATEGORIES); + + values.setValidator((valueMap, status) -> { + String databaseName = valueMap.getString(NAME); + if (StringUtils.isBlank(databaseName)) { + status.setStatusText("Name must be filled in!", MessageType.ERROR); + return false; + } + File directory = valueMap.getFile(DIRECTORY); + if (!directory.isDirectory()) { + status.setStatusText("Invalid directory!", MessageType.ERROR); + return false; + } + File dbFile = new File(directory, databaseName); + File testFile = new File(dbFile.getPath() + BSimServerInfo.H2_FILE_EXTENSION); + if (testFile.exists()) { + status.setStatusText("Database file already exists!", MessageType.ERROR); + return false; + } + return true; + }); + + askValues("Enter Database Parameters", + "Enter values required to create a new BSim H2 database.", values); + + FunctionDatabase h2Database = null; + try { + String databaseName = values.getString(NAME); + File dbDir = values.getFile(DIRECTORY); + String template = values.getChoice(DATABASE_TEMPLATE); + String functionTagsCSV = values.getString(FUNCTION_TAGS); + List tags = parseCSV(functionTagsCSV); + + String exeCatCSV = values.getString(EXECUTABLE_CATEGORIES); + List cats = parseCSV(exeCatCSV); + + File dbFile = new File(dbDir, databaseName); + + BSimServerInfo serverInfo = + new BSimServerInfo(DBType.file, null, 0, dbFile.getAbsolutePath()); + h2Database = BSimClientFactory.buildClient(serverInfo, false); + BSimH2FileDataSource bds = + BSimH2FileDBConnectionManager.getDataSourceIfExists(h2Database.getServerInfo()); + if (bds.getActiveConnections() > 0) { + //if this happens, there is a connection to the database but the + //database file was deleted + Msg.showError(this, null, "Connection Error", + "There is an existing connection to the database!"); + return; + } + + CreateDatabase command = new CreateDatabase(); + command.info = new DatabaseInformation(); + // Put in fields provided on the command line + // If they are null, the template will fill them in + command.info.databasename = databaseName; + command.config_template = template; + command.info.trackcallgraph = true; + ResponseInfo response = command.execute(h2Database); + if (response == null) { + throw new IOException(h2Database.getLastError().message); + } + + for (String tag : tags) { + InstallTagRequest req = new InstallTagRequest(); + req.tag_name = tag; + ResponseInfo resp = req.execute(h2Database); + if (resp == null) { + Error lastError = h2Database.getLastError(); + throw new LSHException(lastError.message); + } + } + + for (String cat : cats) { + InstallCategoryRequest req = new InstallCategoryRequest(); + req.type_name = cat; + ResponseInfo resp = req.execute(h2Database); + if (resp == null) { + Error lastError = h2Database.getLastError(); + throw new LSHException(lastError.message); + } + } + popup("Database " + values.getString(NAME) + " created successfully!"); + } + finally { + if (h2Database != null) { + h2Database.close(); + BSimH2FileDataSource bds = + BSimH2FileDBConnectionManager.getDataSourceIfExists(h2Database.getServerInfo()); + bds.dispose(); + } + } + + } + + //this de-dupes + private List parseCSV(String csv) { + Set parsed = new HashSet<>(); + if (StringUtils.isEmpty(csv)) { + return new ArrayList(); + } + String[] parts = csv.split(","); + for (String p : parts) { + if (!StringUtils.isBlank(p)) { + parsed.add(p.trim()); + } + } + List res = new ArrayList<>(parsed); + res.sort(String::compareTo); + return res; + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/DebugSignatures.java b/Ghidra/Features/BSim/ghidra_scripts/DebugSignatures.java new file mode 100755 index 0000000000..bcd2a2723f --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/DebugSignatures.java @@ -0,0 +1,72 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +import java.util.List; + +import ghidra.app.decompiler.DecompInterface; +import ghidra.app.decompiler.DecompileOptions; +import ghidra.app.decompiler.signature.DebugSignature; +import ghidra.app.script.GhidraScript; +import ghidra.program.model.lang.Language; +import ghidra.program.model.listing.Function; + +public class DebugSignatures extends GhidraScript { + + private static final int SIGNATURE_SETTINGS = 0x45; + + @Override + protected void run() throws Exception { + Function func = this.getFunctionContaining(this.currentAddress); + + if (func == null) { + popup("No function selected!"); + return; + } + + DecompInterface decompiler = new DecompInterface(); + decompiler.setOptions(new DecompileOptions()); + decompiler.toggleSyntaxTree(false); + decompiler.setSignatureSettings(SIGNATURE_SETTINGS); + if (!decompiler.openProgram(this.currentProgram)) { + println("Unable to initalize the Decompiler interface"); + println(decompiler.getLastMessage()); + return; + } + + Language language = this.currentProgram.getLanguage(); + List sigres = decompiler.debugSignatures(func, 10, null); + + StringBuffer buf = new StringBuffer(); + buf.append("\nFunction: "); + buf.append(func.getName()); + buf.append("\nentry: "); + buf.append(func.getEntryPoint().toString()); + buf.append("\n\n"); + if (sigres == null) { + printf("Null sigres!\n"); + } + else { + for (int i = 0; i < sigres.size(); ++i) { + sigres.get(i).printRaw(language, buf); + buf.append("\n"); + } + } + printf("%s\n", buf.toString()); + decompiler.closeProgram(); + decompiler.dispose(); + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/DumpDebugSignatures.py b/Ghidra/Features/BSim/ghidra_scripts/DumpDebugSignatures.py new file mode 100755 index 0000000000..9877b8217d --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/DumpDebugSignatures.py @@ -0,0 +1,61 @@ +## ### +# IP: GHIDRA +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## +# Use the decompiler to generate signatures for the function at the current address, then dump the +# signature hashes and debug information to the console +# @category: BSim.python + +import ghidra.app.decompiler.tracking.DecompInterfaceTracking as DecompInterfaceTracking +import ghidra.app.decompiler.DecompileOptions as DecompileOptions +import generic.lsh.vector.WeightedLSHCosineVectorFactory as WeightedLSHCosineVectorFactory +import ghidra.query.GenSignatures as GenSignatures +import ghidra.xml.NonThreadedXmlPullParserImpl as NonThreadedXmlPullParserImpl +import ghidra.util.xml.SpecXmlUtils as SpecXmlUtils + + +def processFunction(func): + decompiler = DecompInterfaceTracking() + options = DecompileOptions() + decompiler.setOptions(options) + decompiler.toggleSyntaxTree(False) + decompiler.setSignatureSettings(getSettings()) + if not decompiler.openProgram(currentProgram): + print "Unable to initialize the Decompiler interface!" + print "%s" % decompiler.getLastMessage() + return + language = currentProgram.getLanguage() + sigres = decompiler.debugSignatures(func,10,None) + for i,res in enumerate(sigres): + buf = java.lang.StringBuffer() + sigres.get(i).printRaw(language,buf) + print "%s" % buf.toString() + decompiler.closeProgram() + decompiler.dispose() + +def getSettings(): + vectorFactory = WeightedLSHCosineVectorFactory() + id = currentProgram.getLanguageID() + defaultWeightsFile = GenSignatures.getWeightsFile(id,id) + input = defaultWeightsFile.getInputStream() + parser = NonThreadedXmlPullParserImpl(input,"Vector weights parser", SpecXmlUtils.getXmlHandler(),False) + vectorFactory.readWeights(parser) + input.close() + return vectorFactory.getSettings() + +func = currentProgram.getFunctionManager().getFunctionContaining(currentAddress) +if func is None: + print "no function at current address" +else: + processFunction(func) diff --git a/Ghidra/Features/BSim/ghidra_scripts/DumpSignatures.java b/Ghidra/Features/BSim/ghidra_scripts/DumpSignatures.java new file mode 100755 index 0000000000..aad4ddc12b --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/DumpSignatures.java @@ -0,0 +1,115 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Use the decompiler to generate signatures for the function currently containing the cursor +// and dump the signature hashes to the console +//@category BSim + +import java.io.*; +import java.util.List; + +import org.xml.sax.SAXException; + +import generic.jar.ResourceFile; +import generic.lsh.vector.LSHVectorFactory; +import generic.lsh.vector.WeightedLSHCosineVectorFactory; +import ghidra.app.decompiler.DecompInterface; +import ghidra.app.decompiler.DecompileOptions; +import ghidra.app.decompiler.signature.DebugSignature; +import ghidra.app.decompiler.signature.SignatureResult; +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.query.GenSignatures; +import ghidra.program.model.lang.Language; +import ghidra.program.model.lang.LanguageID; +import ghidra.program.model.listing.Function; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +public class DumpSignatures extends GhidraScript { + + private LSHVectorFactory vectorFactory; + + @Override + public void run() throws Exception { + Function func = this.getFunctionContaining(this.currentAddress); + if (func == null) { + return; + } + buildLSHVectorFactory(); + boolean debug = false; + DecompInterface decompiler = new DecompInterface(); + decompiler.setOptions(new DecompileOptions()); + decompiler.setSignatureSettings(vectorFactory.getSettings()); + decompiler.toggleSyntaxTree(false); + if (!decompiler.openProgram(this.currentProgram)) { + println("Unable to initalize the Decompiler interface"); + println(decompiler.getLastMessage()); + return; + } + if (!debug) { + SignatureResult sigres = decompiler.generateSignatures(func, false, 10, null); + StringBuffer buf = new StringBuffer("\n"); + for (int feature : sigres.features) { + buf.append(Integer.toHexString(feature)); + buf.append("\n"); + } + println(buf.toString()); + } + else { + Language language = this.currentProgram.getLanguage(); + List sigres = decompiler.debugSignatures(func, 10, null); + StringBuffer buf = new StringBuffer("\n"); + for (int i = 0; i < sigres.size(); ++i) { + sigres.get(i).printRaw(language, buf); + buf.append("\n"); + } + println(buf.toString()); + } + decompiler.closeProgram(); + decompiler.dispose(); + } + + private static void readWeights(LSHVectorFactory vectorFactory, ResourceFile weightsFile) + throws FileNotFoundException, IOException, SAXException { + InputStream input = weightsFile.getInputStream(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(input, "Vector weights parser", + SpecXmlUtils.getXmlHandler(), false); + vectorFactory.readWeights(parser); + input.close(); + } + + private void buildLSHVectorFactory() { + vectorFactory = new WeightedLSHCosineVectorFactory(); + try { + LanguageID id = currentProgram.getLanguageID(); + ResourceFile defaultWeightsFile = GenSignatures.getWeightsFile(id, id); + readWeights(vectorFactory, defaultWeightsFile); + } + catch (FileNotFoundException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + catch (IOException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + catch (SAXException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/DumpSignatures.py b/Ghidra/Features/BSim/ghidra_scripts/DumpSignatures.py new file mode 100755 index 0000000000..1f336303aa --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/DumpSignatures.py @@ -0,0 +1,61 @@ +## ### +# IP: GHIDRA +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## +# Use the decompiler to generate signatures for the function at the current address, then dump the +# signature hashes to the console +# @category: BSim.python + +import ghidra.app.decompiler.tracking.DecompInterfaceTracking as DecompInterfaceTracking +import ghidra.app.decompiler.DecompileOptions as DecompileOptions +import generic.lsh.vector.WeightedLSHCosineVectorFactory as WeightedLSHCosineVectorFactory +import ghidra.query.GenSignatures as GenSignatures +import ghidra.xml.NonThreadedXmlPullParserImpl as NonThreadedXmlPullParserImpl +import ghidra.util.xml.SpecXmlUtils as SpecXmlUtils + + +def processFunction(func): + decompiler = ghidra.app.decompiler.tracking.DecompInterfaceTracking() + options = ghidra.app.decompiler.DecompileOptions() + decompiler.setOptions(options) + decompiler.toggleSyntaxTree(False) + decompiler.setSignatureSettings(getSettings()) + if not decompiler.openProgram(currentProgram): + print "Unable to initialize the Decompiler interface!" + print "%s" % decompiler.getLastMessage() + return + sigres = decompiler.generateSignatures(func, False, 10, None) + buf = java.lang.StringBuffer() + for i,res in enumerate(sigres.features): + buf.append(java.lang.Integer.toHexString(sigres.features[i])) + buf.append("\n") + print buf.toString() + decompiler.closeProgram() + decompiler.dispose() + +def getSettings(): + vectorFactory = WeightedLSHCosineVectorFactory() + id = currentProgram.getLanguageID() + defaultWeightsFile = GenSignatures.getWeightsFile(id,id) + input = defaultWeightsFile.getInputStream() + parser = NonThreadedXmlPullParserImpl(input,"Vector weights parser", SpecXmlUtils.getXmlHandler(),False) + vectorFactory.readWeights(parser) + input.close() + return vectorFactory.getSettings() + +func = currentProgram.getFunctionManager().getFunctionContaining(currentAddress) +if func is None: + print "no function at current address" +else: + processFunction(func) diff --git a/Ghidra/Features/BSim/ghidra_scripts/ExampleOverviewQuery.java b/Ghidra/Features/BSim/ghidra_scripts/ExampleOverviewQuery.java new file mode 100755 index 0000000000..6b510930ef --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/ExampleOverviewQuery.java @@ -0,0 +1,69 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +//Example of how to perform an overview query in a script. +//@category BSim +import java.util.HashSet; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.query.facade.SFOverviewInfo; +import ghidra.features.bsim.query.facade.SimilarFunctionQueryService; +import ghidra.features.bsim.query.protocol.ResponseNearestVector; +import ghidra.features.bsim.query.protocol.SimilarityVectorResult; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.listing.*; + + +public class ExampleOverviewQuery extends GhidraScript { + private static final double SIMILARITY_BOUND = 0.7; + private static final double SIGNIFICANCE_BOUND = 0.0; + + + @Override + protected void run() throws Exception { + Program queryingProgram = currentProgram; + HashSet funcsToQuery = new HashSet<>(); + FunctionIterator fIter = queryingProgram.getFunctionManager().getFunctionsNoStubs(true); + for (Function func : fIter){ + funcsToQuery.add((FunctionSymbol) func.getSymbol()); + } + SFOverviewInfo overviewInfo = new SFOverviewInfo(funcsToQuery); + overviewInfo.setSimilarityThreshold(SIMILARITY_BOUND); + overviewInfo.setSignificanceThreshold(SIGNIFICANCE_BOUND); + + try (SimilarFunctionQueryService queryService = + new SimilarFunctionQueryService(queryingProgram)) { + String DATABASE_URL = askString("Enter database URL", "URL:"); + queryService.initializeDatabase(DATABASE_URL); + LSHVectorFactory vectorFactory = queryService.getLSHVectorFactory(); + + ResponseNearestVector overviewResults = + queryService.overviewSimilarFunctions(overviewInfo, null, monitor); + StringBuilder buf = new StringBuilder(); + buf.append("\n"); + for (SimilarityVectorResult result : overviewResults.result) { + buf.append("Name: ").append(result.getBase().getFunctionName()).append("\n"); + buf.append("Hit Count: ").append(result.getTotalCount()).append("\n"); + buf.append("Self-significance: "); + buf.append(vectorFactory + .getSelfSignificance(result.getBase().getSignatureRecord().getLSHVector())); + buf.append("\n\n"); + } + printf("%s\n", buf.toString()); + } + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/ExampleOverviewQuery.py b/Ghidra/Features/BSim/ghidra_scripts/ExampleOverviewQuery.py new file mode 100755 index 0000000000..5a7f4751d3 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/ExampleOverviewQuery.py @@ -0,0 +1,47 @@ +## ### +# IP: GHIDRA +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## +# Example of how to perform an overview query in a script +# @category BSim.python + +import ghidra.query.facade.SFOverviewInfo as SFOverviewInfo +import ghidra.query.facade.SimilarFunctionQueryService as SimilarFunctionQueryService +import java.util.HashSet + +SIMILARITY_BOUND = 0.7 +SIGNIFICANCE_BOUND = 0.0 + +funcsToQuery = java.util.HashSet() +fIter = currentProgram.getFunctionManager().getFunctionsNoStubs(True) +for func in fIter: + funcsToQuery.add(func.getSymbol()) + +overviewInfo = SFOverviewInfo(funcsToQuery) +overviewInfo.setSimilarityThreshold(SIMILARITY_BOUND) +overviewInfo.setSignificanceThreshold(SIGNIFICANCE_BOUND) + +queryService = SimilarFunctionQueryService(currentProgram) +DB_URL = askString("Enter database URL", "URL:") +queryService.initializeDatabase(DB_URL) +vectorFactory = queryService.getLSHVectorFactory() + +overviewResults = queryService.overviewSimilarFunctions(overviewInfo, monitor) + +for result in overviewResults.result: + print "Name: %s" % result.getBase().getFunctionName() + print "Hit Count: %d" % result.getTotalCount() + print "Self-significance: %f\n" % vectorFactory.getSelfSignificance(result.getBase().getSignatureRecord().getLSHVector()) + +queryService.dispose() diff --git a/Ghidra/Features/BSim/ghidra_scripts/ExampleQueryClient.java b/Ghidra/Features/BSim/ghidra_scripts/ExampleQueryClient.java new file mode 100755 index 0000000000..5ee7eab373 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/ExampleQueryClient.java @@ -0,0 +1,83 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Example of connecting to a BSim server and requesting executable and function records +//@category BSim + +import java.io.StringWriter; +import java.net.URL; +import java.util.List; + +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.query.BSimClientFactory; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.*; +import ghidra.util.Msg; + +public class ExampleQueryClient extends GhidraScript { + + @Override + protected void run() throws Exception { + URL url = BSimClientFactory.deriveBSimURL("ghidra://localhost/repo"); + try (FunctionDatabase client = BSimClientFactory.buildClient(url, false)) { + if (!client.initialize()) { + Msg.error(this, "Unable to connect to server"); + return; + } + + QueryInfo query = new QueryInfo(); + ResponseInfo resp = query.execute(client); + StringWriter write = new StringWriter(); + resp.saveXml(write); + write.flush(); + + QueryName exequery = new QueryName(); + exequery.spec.exename = "libdocdoxygenplugin.so"; + ResponseName respname = exequery.execute(client); + if (respname == null) { + Msg.error(this, client.getLastError()); + return; + } + ExecutableRecord erec = respname.manage.getExecutableRecordSet().first(); + FunctionDescription funcrec = + respname.manage.findFunctionByName("DocDoxygenPlugin::createCatalog", erec); + + QueryChildren childquery = new QueryChildren(); + childquery.md5sum = funcrec.getExecutableRecord().getMd5(); + childquery.functionKeys.add(new FunctionEntry(funcrec)); + + ResponseChildren respchild = childquery.execute(client); + if (respchild == null) { + Msg.error(this, client.getLastError()); + return; + } + for (int i = 0; i < respchild.correspond.size(); ++i) { + FunctionDescription func = respchild.correspond.get(i); + List callgraphRecord = func.getCallgraphRecord(); + if (callgraphRecord != null) { + for (int j = 0; j < callgraphRecord.size(); ++j) { + write.write( + callgraphRecord.get(j).getFunctionDescription().getFunctionName()); + write.write('\n'); + } + } + } + write.flush(); + Msg.info(this, write.toString()); + } + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/GenerateSignatures.java b/Ghidra/Features/BSim/ghidra_scripts/GenerateSignatures.java new file mode 100755 index 0000000000..ec28acfebc --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/GenerateSignatures.java @@ -0,0 +1,73 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Generate signatures for every function in the current executable and write in XML form to +// a user specified file. +//@category BSim + +import java.io.*; +import java.util.Iterator; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.GenSignatures; +import ghidra.features.bsim.query.client.Configuration; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.FunctionManager; + +public class GenerateSignatures extends GhidraScript { + + @Override + public void run() throws Exception { + final String md5string = currentProgram.getExecutableMD5(); + if ((md5string == null) || (md5string.length() < 10)) { + throw new IOException("Could not get MD5 on file: " + currentProgram.getName()); + } + final String basename = "sigs_" + md5string; + System.setProperty("ghidra.output", basename); // Inform parallel controller of output name + File file = null; + // This form of askString will work for both standalone execution or for parallel + final File workingdir = askDirectory("GenerateSignatures:", "Working directory"); + if (!workingdir.isDirectory()) { + popup("Must select a working directory!"); + return; + } + file = new File(workingdir, basename); + + final LSHVectorFactory vectorFactory = FunctionDatabase.generateLSHVectorFactory(); + final GenSignatures gensig = new GenSignatures(true); + final String templatename = + askString("GenerateSignatures:", "Database template", "medium_nosize"); + final Configuration config = FunctionDatabase.loadConfigurationTemplate(templatename); + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + gensig.setVectorFactory(vectorFactory); + gensig.addExecutableCategories(config.info.execats); + gensig.addFunctionTags(config.info.functionTags); + gensig.addDateColumnName(config.info.dateColumnName); + final String repo = "ghidra://localhost/" + state.getProject().getName(); + final String path = GenSignatures.getPathFromDomainFile(currentProgram); + gensig.openProgram(this.currentProgram, null, null, null, repo, path); + final FunctionManager fman = currentProgram.getFunctionManager(); + final Iterator iter = fman.getFunctions(true); + gensig.scanFunctions(iter, fman.getFunctionCount(), monitor); + final FileWriter fwrite = new FileWriter(file); + final DescriptionManager manager = gensig.getDescriptionManager(); + manager.saveXml(fwrite); + fwrite.close(); + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/GenerateSignatures.py b/Ghidra/Features/BSim/ghidra_scripts/GenerateSignatures.py new file mode 100755 index 0000000000..becc8233ce --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/GenerateSignatures.py @@ -0,0 +1,58 @@ +## ### +# IP: GHIDRA +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## +#Generate signatures for every function in the current program and write them to an XML file in a user-specified directory +#@category BSim.python + +import java.lang.System as System +import java.io.File as File +import ghidra.query.FunctionDatabase as FunctionDatabase +import ghidra.query.GenSignatures as GenSignatures +import java.io.FileWriter as FileWriter + +def run(): + md5String = currentProgram.getExecutableMD5() + if (md5String is None) or (len(md5String) < 10): + raise IOException("Could not get MD5 on file: " + currentProgram.getName()) + basename = "sigs_" + md5String + System.setProperty("ghidra.output",basename) + workingDir = askDirectory("GenerateSignatures:", "Working Directory") + if not workingDir.isDirectory(): + popup("Must select a working directory") + return + outfile = File(workingDir,basename) + vectorFactory = FunctionDatabase.generateLSHVectorFactory() + gensig = GenSignatures(True) + templateName = askString("GenerateSignatures:", "Database template", "medium_nosize") + config = FunctionDatabase.loadConfigurationTemplate(templateName) + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings) + gensig.setVectorFactory(vectorFactory) + gensig.addExecutableCategories(config.info.execats) + gensig.addFunctionTags(config.info.functionTags) + gensig.addDateColumnName(config.info.dateColumnName) + repo = "ghidra://localhost/" + state.getProject().getName() + path = GenSignatures.getPathFromDomainFile(currentProgram) + gensig.openProgram(currentProgram,None,None,None,repo,path) + fman = currentProgram.getFunctionManager() + iter = fman.getFunctions(True) + gensig.scanFunctions(iter, fman.getFunctionCount(), monitor) + fwrite = FileWriter(outfile) + manager = gensig.getDescriptionManager() + manager.saveXml(fwrite) + fwrite.close() + return + +run() + diff --git a/Ghidra/Features/BSim/ghidra_scripts/LocalBSimQueryScript.java b/Ghidra/Features/BSim/ghidra_scripts/LocalBSimQueryScript.java new file mode 100644 index 0000000000..2523dcae72 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/LocalBSimQueryScript.java @@ -0,0 +1,443 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +//Queries all functions in the current selection (or all functions in the current program if +//the current selection is null) against all functions in a user-selected program. +//@category BSim + +import java.util.*; + +import org.apache.commons.collections4.IteratorUtils; + +import generic.lsh.vector.*; +import ghidra.app.decompiler.DecompileException; +import ghidra.app.plugin.core.functioncompare.FunctionComparisonProvider; +import ghidra.app.script.GhidraScript; +import ghidra.app.services.FunctionComparisonService; +import ghidra.app.tablechooser.*; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.client.Configuration; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.*; + +//TODO: docs + +public class LocalBSimQueryScript extends GhidraScript { + + //functions with self significance below this bound will be skipped + private static final double SELF_SIGNIFICANCE_BOUND = 15.0; + //bsim database template determining the signature settings + private static final String TEMPLATE_NAME = "medium_nosize"; + //these are analogous to the bounds in a bsim query + private static final double MATCH_SIMILARITY_LOWER_BOUND = 0.0; + private static final double MATCH_CONFIDENCE_LOWER_BOUND = 0.0; + private static final int MATCHES_PER_FUNCTION = 10; + //decrease this if you only want to see matches that aren't exact + //for instance, when looking for changes between two versions of a program + private static final double MATCH_SIMILARITY_UPPER_BOUND = 1.0; + + private TableChooserDialog tableDialog; + + @Override + protected void run() throws Exception { + if (isRunningHeadless()) { + popup("This script cannot be run headlessly."); + return; + } + + Set sourceFuncs = new HashSet<>(); + if (currentSelection == null) { + IteratorUtils.forEach(currentProgram.getFunctionManager().getFunctions(true), + x -> sourceFuncs.add(x)); + } + else { + IteratorUtils.forEach( + currentProgram.getFunctionManager().getFunctionsOverlapping(currentSelection), + x -> sourceFuncs.add(x)); + } + + if (sourceFuncs.isEmpty()) { + this.popup("No non-stub functions to query!"); + return; + } + + Program targetProgram = askProgram("Select Target Program"); + if (targetProgram == null) { + return; + } + try { + List localMatches = null; + + //use special optimized method when the target program is the same as the current program + //in that case, a given function might be in both the source and target sets + //but we only want to generate signatures for it once + if (currentProgram.getUniqueProgramID() == targetProgram.getUniqueProgramID()) { + localMatches = getMatchesCurrentProgram(sourceFuncs); + } + else { + //in this case there is no overlap between the source and target functions + localMatches = getMatchesTwoPrograms(sourceFuncs, currentProgram, targetProgram); + } + if (localMatches.isEmpty()) { + popup("No matches meeting criteria."); + return; + } + Collections.sort(localMatches); + initializeTable(currentProgram, targetProgram); + + //again, use an optimized method for the special case when target program is the same + //as the current program + if (currentProgram.getUniqueProgramID() == targetProgram.getUniqueProgramID()) { + addMatchesOneProgram(localMatches, sourceFuncs); + } + else { + addMatchesTwoPrograms(localMatches); + } + } + finally { + targetProgram.release(this); + } + } + + /** + * Iterate through the list of sorted matches, adding the top MATCHES_PER_FUNCTION elements + * to the table for each source function. + * @param localMatches matches in decreasing order of confidence + */ + private void addMatchesTwoPrograms(List localMatches) { + Map matchCounts = new HashMap<>(); + for (LocalBSimMatch match : localMatches) { + int count = matchCounts.getOrDefault(match.getSourceFunc(), 0); + if (count >= MATCHES_PER_FUNCTION) { + continue; + } + tableDialog.add(match); + matchCounts.put(match.getSourceFunc(), count + 1); + } + } + + /** + * Iterate through the list of sorted matches, adding the top MATCHES_PER_FUNCTION elements + * to the table for each function ins {@code sourceFuncSet}. + * + * By construction, the matches in this list have the "source" function before the "target" + * function (in address order). This is an optimization to prevent essentially the same + * data from appearing in the list twice (since the BSim similarity and confidence operations + * are commutative). So, for each match, we need to check whether the source or the + * target are in {@code sourceFuncSet}. + * + * @param localMatches matches in decreasing order of confidence + * @param sourceFuncSet source functions + */ + private void addMatchesOneProgram(List localMatches, + Set sourceFuncSet) { + Map matchCounts = new HashMap<>(); + for (LocalBSimMatch match : localMatches) { + Function leftFunc = match.getSourceFunc(); + int leftCount = matchCounts.getOrDefault(leftFunc, 0); + if (sourceFuncSet.contains(leftFunc) && leftCount < MATCHES_PER_FUNCTION) { + tableDialog.add(match); + matchCounts.put(leftFunc, leftCount + 1); + } + Function rightFunc = match.getTargetFunc(); + int rightCount = matchCounts.getOrDefault(rightFunc, 0); + if (sourceFuncSet.contains(rightFunc) && rightCount < MATCHES_PER_FUNCTION) { + LocalBSimMatch switched = new LocalBSimMatch(rightFunc, leftFunc, + match.getSimilarity(), match.getSignificance()); + tableDialog.add(switched); + matchCounts.put(rightFunc, rightCount + 1); + } + } + } + + private List getMatchesCurrentProgram(Set funcs) + throws LSHException, DecompileException { + List bsimMatches = new ArrayList<>(); + LSHVectorFactory vectorFactory = getVectorFactory(); + + //generate the signatures for *all* functions in the program... + FunctionManager fman = currentProgram.getFunctionManager(); + Iterator iter = fman.getFunctions(true); + GenSignatures gensig = + generateSignatures(currentProgram, iter, fman.getFunctionCount(), vectorFactory); + + //...but use sourceFuncAddrs to ensure that source functions are in the + //funcs set + Set sourceFuncAddrs = new HashSet<>(); + for (Function func : funcs) { + sourceFuncAddrs.add(func.getEntryPoint().getOffset()); + } + Iterator sourceDescripts = + gensig.getDescriptionManager().listAllFunctions(); + VectorCompare vecCompare = new VectorCompare(); + while (sourceDescripts.hasNext()) { + FunctionDescription srcDesc = sourceDescripts.next(); + //skip if not in selection + if (!sourceFuncAddrs.contains(srcDesc.getAddress())) { + continue; + } + //skip if self-significance too small + LSHVector srcVector = srcDesc.getSignatureRecord().getLSHVector(); + if (vectorFactory.getSelfSignificance(srcVector) <= SELF_SIGNIFICANCE_BOUND) { + continue; + } + Iterator targetDescripts = + gensig.getDescriptionManager().listAllFunctions(); + Function srcFunc = getFunction(currentProgram, srcDesc.getAddress()); + while (targetDescripts.hasNext()) { + //skip if target before srcFunc in address order + //AND target is one of the source functions (i.e., in funcs) + FunctionDescription targetDesc = targetDescripts.next(); + long targetAddress = targetDesc.getAddress(); + if (sourceFuncAddrs.contains(targetAddress) && + targetAddress <= srcDesc.getAddress()) { + continue; + } + //skip if self-significance too small + LSHVector targetVector = targetDesc.getSignatureRecord().getLSHVector(); + if (vectorFactory.getSelfSignificance(targetVector) <= SELF_SIGNIFICANCE_BOUND) { + continue; + } + double sim = srcVector.compare(targetVector, vecCompare); + double sig = vectorFactory.calculateSignificance(vecCompare); + if (sig >= MATCH_CONFIDENCE_LOWER_BOUND && MATCH_SIMILARITY_LOWER_BOUND <= sim && + sim <= MATCH_SIMILARITY_UPPER_BOUND) { + Function targetFunc = getFunction(currentProgram, targetDesc.getAddress()); + bsimMatches.add(new LocalBSimMatch(srcFunc, targetFunc, sim, sig)); + } + } + } + return bsimMatches; + } + + private List getMatchesTwoPrograms(Set srcFuncs, + Program sourceProgram, Program targetProgram) throws LSHException, DecompileException { + List bsimMatches = new ArrayList<>(); + LSHVectorFactory vectorFactory = getVectorFactory(); + GenSignatures srcSigs = + generateSignatures(sourceProgram, srcFuncs.iterator(), srcFuncs.size(), vectorFactory); + FunctionManager targetFuncMan = targetProgram.getFunctionManager(); + Iterator targetFuncIter = targetFuncMan.getFunctions(true); + GenSignatures targetSigs = generateSignatures(targetProgram, targetFuncIter, + targetFuncMan.getFunctionCount(), vectorFactory); + Iterator sourceDescripts = + srcSigs.getDescriptionManager().listAllFunctions(); + VectorCompare vecCompare = new VectorCompare(); + while (sourceDescripts.hasNext()) { + FunctionDescription srcDesc = sourceDescripts.next(); + //skip if self-significance too small + LSHVector srcVector = srcDesc.getSignatureRecord().getLSHVector(); + if (vectorFactory.getSelfSignificance(srcVector) <= SELF_SIGNIFICANCE_BOUND) { + continue; + } + Iterator targetDescripts = + targetSigs.getDescriptionManager().listAllFunctions(); + Function srcFunc = getFunction(sourceProgram, srcDesc.getAddress()); + while (targetDescripts.hasNext()) { + FunctionDescription targetDesc = targetDescripts.next(); + //skip if self-significance too small + LSHVector targetVector = targetDesc.getSignatureRecord().getLSHVector(); + if (vectorFactory.getSelfSignificance(targetVector) <= SELF_SIGNIFICANCE_BOUND) { + continue; + } + double sim = srcVector.compare(targetVector, vecCompare); + double sig = vectorFactory.calculateSignificance(vecCompare); + if (sig >= MATCH_CONFIDENCE_LOWER_BOUND && MATCH_SIMILARITY_LOWER_BOUND <= sim && + sim <= MATCH_SIMILARITY_UPPER_BOUND) { + Function targetFunc = getFunction(targetProgram, targetDesc.getAddress()); + bsimMatches.add(new LocalBSimMatch(srcFunc, targetFunc, sim, sig)); + } + } + } + return bsimMatches; + } + + private Function getFunction(Program program, long offset) { + Address addr = program.getAddressFactory().getDefaultAddressSpace().getAddress(offset); + return program.getFunctionManager().getFunctionAt(addr); + } + + private LSHVectorFactory getVectorFactory() throws LSHException { + LSHVectorFactory vectorFactory = FunctionDatabase.generateLSHVectorFactory(); + Configuration config = FunctionDatabase.loadConfigurationTemplate(TEMPLATE_NAME); + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + return vectorFactory; + } + + private GenSignatures generateSignatures(Program program, Iterator funcs, int count, + LSHVectorFactory vectorFactory) throws LSHException, DecompileException { + GenSignatures gensig = new GenSignatures(false); + gensig.setVectorFactory(vectorFactory); + gensig.openProgram(program, null, null, null, null, null); + gensig.scanFunctions(funcs, count, monitor); + return gensig; + } + + class LocalBSimMatch implements Comparable, AddressableRowObject { + private Function sourceFunc; + private Function targetFunc; + private double similarity; + private double significance; + + public LocalBSimMatch(Function sourceFunc, Function targetFunc, double sim, double signif) { + this.sourceFunc = sourceFunc; + this.targetFunc = targetFunc; + this.similarity = sim; + this.significance = signif; + } + + public Function getSourceFunc() { + return sourceFunc; + } + + public Function getTargetFunc() { + return targetFunc; + } + + public double getSimilarity() { + return similarity; + } + + public double getSignificance() { + return significance; + } + + public Program getSourceProgram() { + return sourceFunc.getProgram(); + } + + public Program getTargetProgram() { + return targetFunc.getProgram(); + } + + @Override + public int compareTo(LocalBSimQueryScript.LocalBSimMatch o) { + return -Double.compare(significance, o.significance); + } + + @Override + public Address getAddress() { + return sourceFunc.getEntryPoint(); + } + } + + /**************************************************************************************** + * table stuff + ****************************************************************************************/ + + class CompareMatchesExecutor implements TableChooserExecutor { + + private FunctionComparisonService compareService; + private FunctionComparisonProvider comparisonProvider; + + public CompareMatchesExecutor() { + compareService = state.getTool().getService(FunctionComparisonService.class); + } + + @Override + public String getButtonName() { + return "Compare Selected Matches"; + } + + @Override + public boolean execute(AddressableRowObject rowObject) { + LocalBSimMatch match = (LocalBSimMatch) rowObject; + if (comparisonProvider == null) { + comparisonProvider = + compareService.compareFunctions(match.getSourceFunc(), match.getTargetFunc()); + } + else { + compareService.compareFunctions(match.getSourceFunc(), match.getTargetFunc(), + comparisonProvider); + } + return false; + } + } + + private void initializeTable(Program sourceProgram, Program targetProgram) { + StringBuilder titleBuilder = new StringBuilder("Local BSim Matches: "); + titleBuilder.append(sourceProgram.getDomainFile().getPathname()); + titleBuilder.append(" -> "); + titleBuilder.append(targetProgram.getDomainFile().getPathname()); + tableDialog = + createTableChooserDialog(titleBuilder.toString(), new CompareMatchesExecutor()); + configureTableColumns(tableDialog); + tableDialog.setMinimumSize(800, 400); + tableDialog.show(); + tableDialog.setMessage(null); + } + + private void configureTableColumns(TableChooserDialog dialog) { + + ColumnDisplay simColumn = new AbstractComparableColumnDisplay() { + + @Override + public Double getColumnValue(AddressableRowObject rowObject) { + return ((LocalBSimMatch) rowObject).getSimilarity(); + } + + @Override + public String getColumnName() { + return "Similarity"; + } + }; + + ColumnDisplay sigColumn = new AbstractComparableColumnDisplay() { + + @Override + public Double getColumnValue(AddressableRowObject rowObject) { + return ((LocalBSimMatch) rowObject).getSignificance(); + } + + @Override + public String getColumnName() { + return "Significance"; + } + }; + + StringColumnDisplay sourceFuncColumn = new StringColumnDisplay() { + + @Override + public String getColumnValue(AddressableRowObject rowObject) { + return ((LocalBSimMatch) rowObject).getSourceFunc().getName(true); + } + + @Override + public String getColumnName() { + return "Source Function"; + } + }; + + StringColumnDisplay targetFuncColumn = new StringColumnDisplay() { + + @Override + public String getColumnValue(AddressableRowObject rowObject) { + return ((LocalBSimMatch) rowObject).getTargetFunc().getName(true); + } + + @Override + public String getColumnName() { + return "Target Function"; + } + }; + + dialog.addCustomColumn(simColumn); + dialog.addCustomColumn(sigColumn); + dialog.addCustomColumn(sourceFuncColumn); + dialog.addCustomColumn(targetFuncColumn); + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/QueryFunction.java b/Ghidra/Features/BSim/ghidra_scripts/QueryFunction.java new file mode 100755 index 0000000000..2b69dc680b --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/QueryFunction.java @@ -0,0 +1,108 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Example of querying a BSim database about a single function +//@category BSim + +import java.net.URL; +import java.util.Iterator; + +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.*; +import ghidra.program.model.listing.Function; + + +public class QueryFunction extends GhidraScript { + + //GenSignatures gensig; + //FunctionDatabase database; + private static final int MATCHES_PER_FUNC = 10; + private static final double SIMILARITY_BOUND = 0.7; + private static final double CONFIDENCE_BOUND = 0.0; + + @Override + public void run() throws Exception { + if (currentProgram == null) { + return; + } + Function func = this.getFunctionContaining(this.currentAddress); + if (func == null){ + popup("No function selected!"); + return; + } + + String DATABASE_URL = askString("Enter Database URL", "URL"); + URL url = BSimClientFactory.deriveBSimURL(DATABASE_URL); + try (FunctionDatabase database = BSimClientFactory.buildClient(url, false)) { + if (!database.initialize()) { + println(database.getLastError().message); + return; + } + + GenSignatures gensig = new GenSignatures(false); + try { + gensig.setVectorFactory(database.getLSHVectorFactory()); + gensig.openProgram(currentProgram, null, null, null, null, null); + + DescriptionManager manager = gensig.getDescriptionManager(); + gensig.scanFunction(func); + + QueryNearest query = new QueryNearest(); + query.manage = manager; + query.max = MATCHES_PER_FUNC; + query.thresh = SIMILARITY_BOUND; + query.signifthresh = CONFIDENCE_BOUND; + + ResponseNearest response = query.execute(database); + if (response == null) { + println(database.getLastError().message); + return; + } + Iterator iter = response.result.iterator(); + StringBuffer buf = new StringBuffer(); + while (iter.hasNext()) { + SimilarityResult sim = iter.next(); + FunctionDescription base = sim.getBase(); + ExecutableRecord exe = base.getExecutableRecord(); + buf.append("\nExecutable: ") + .append(exe.getNameExec()) + .append("\nFunction: ") + .append(base.getFunctionName()) + .append('\n'); + Iterator subiter = sim.iterator(); + while (subiter.hasNext()) { + SimilarityNote note = subiter.next(); + FunctionDescription fdesc = note.getFunctionDescription(); + ExecutableRecord exerec = fdesc.getExecutableRecord(); + buf.append(" Executable: "); + buf.append(exerec.getNameExec()) + .append("\n Matching Function name: ") + .append(fdesc.getFunctionName()); + buf.append("\n Similarity: ").append(note.getSimilarity()); + buf.append("\n Significance: ").append(note.getSignificance()); + buf.append("\n\n"); + } + } + println(buf.toString()); + } + finally { + gensig.dispose(); + } + } + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/QueryFunction.py b/Ghidra/Features/BSim/ghidra_scripts/QueryFunction.py new file mode 100755 index 0000000000..38a16768b6 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/QueryFunction.py @@ -0,0 +1,78 @@ +## ### +# IP: GHIDRA +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## +# Example of performing a BSim query on a single function +# @category BSim.python + +import ghidra.query.BSimClientFactory as BSimClientFactory +import ghidra.query.GenSignatures as GenSignatures +import ghidra.query.protocol.QueryNearest as QueryNearest + +MATCHES_PER_FUNC = 100 +SIMILARITY_BOUND = 0.7 +CONFIDENCE_BOUND = 0.0 + +def query(func): + DATABASE_URL = askString("Enter Database URL", "URL") + url = BSimClientFactory.deriveBSimURL(DATABASE_URL) + database = BSimClientFactory.buildClient(url,False) + if not database.initialize(): + print database.getLastError().message + return + gensig = GenSignatures(False) + gensig.setVectorFactory(database.getLSHVectorFactory()) + gensig.openProgram(currentProgram,None,None,None,None,None) + + gensig.scanFunction(func) + + query = QueryNearest() + query.manage = gensig.getDescriptionManager() + query.max = MATCHES_PER_FUNC + query.thresh = SIMILARITY_BOUND + query.signifthresh = CONFIDENCE_BOUND + + response = database.query(query) + if response is None: + print database.getLastError().message + return + simIter = response.result.iterator() + while simIter.hasNext(): + sim = simIter.next() + base = sim.getBase() + exe = base.getExecutableRecord() + print "Source executable: %s; source function: %s" % (exe.getNameExec(),base.getFunctionName()) + subIter = sim.iterator() + while subIter.hasNext(): + note = subIter.next() + fdesc = note.getFunctionDescription() + exerec = fdesc.getExecutableRecord() + print " Executable: %s" % exerec.getNameExec() + print " Matching Function name: %s " % fdesc.getFunctionName() + print " Similarity: %f" % note.getSimilarity() + print " Significance: %f\n" % note.getSignificance() + gensig.dispose() + database.close() + return; + +if currentProgram is None: + popup("currentProgram is None!") +else: + func = currentProgram.getFunctionManager().getFunctionContaining(currentAddress) + if func is None: + popup("Cursor must be in a function!") + else: + query(func) + + diff --git a/Ghidra/Features/BSim/ghidra_scripts/QueryWithFiltersScript.java b/Ghidra/Features/BSim/ghidra_scripts/QueryWithFiltersScript.java new file mode 100755 index 0000000000..508f2423dc --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/QueryWithFiltersScript.java @@ -0,0 +1,333 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +//Example of a script to perform a more involved BSim query. +//@category BSim +import java.util.*; +import java.util.function.BiPredicate; + +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.gui.filters.*; +import ghidra.features.bsim.gui.search.results.BSimMatchResult; +import ghidra.features.bsim.gui.search.results.ExecutableResult; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.FunctionDatabase.ErrorCategory; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.features.bsim.query.facade.*; +import ghidra.features.bsim.query.protocol.BSimFilter; +import ghidra.features.bsim.query.protocol.PreFilter; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.SourceType; +import ghidra.util.exception.CancelledException; + +/** + * Script showing how to apply filters to a BSim query. Currently we support three types + * of filters, described below: + * + * 1. QUERY THRESHOLDS + * These are the items at the top of the BSim query dialog: + * Similarity + * Confidence + * Matches per Function + * These are server-side filters that will be applied when the db is queried. + * + * 2. PREFILTERS + * Allows users to identify functions that meet certain criteria by specifying + * {@link BiPredicate}s. Any functions matching the predicate(s) will be included + * in the result set. + * + * 3. EXECUTABLE FILTERS + * These are predefined filters that can be applied on the server or on the + * client (applied only to the results of a query). On the BSim query + * dialog these are the items in the filter pulldown menu. + * @see BSimFilterType + * + * SCRIPT FLOW + * This example script does the following: + * + * 1) Set threshold filters + * 2) Set prefilters + * 3) Set executable filters + * 4) Query the database & print results + * 5) Set new executable filters + * 6) Print results + * + * NOTES: 1. You will be queried for the location of the BSim database. This URL + * will take the form "ghidra:/// + * + * 2. This script is only an example - the specific filters demonstrated + * here will not necessarily apply to what's in your BSim database. + * + */ +public class QueryWithFiltersScript extends GhidraScript { + + // Threshold settings. + private static final int MAX_NUM_FUNCTIONS = 100; + private static final double SIMILARITY_BOUND = 0.7; + private static final double SIGNIFICANCE_BOUND = 0.0; + + // Restricts the number of results. + private static final int NUM_EXES_TO_DISPLAY = 10; + + // Prefilter value we'll be setting. + private static final double SELF_SIGNIFICANCE_BOUND = 40.0; + + private HashSet funcsToQuery; + private SimilarFunctionQueryService queryService; + private SFQueryInfo queryInfo; + private BSimFilter bsimFilter; + + @Override + protected void run() throws Exception { + + funcsToQuery = getFunctionsToQuery(currentProgram); + queryService = new SimilarFunctionQueryService(currentProgram); + queryInfo = new SFQueryInfo(funcsToQuery); + bsimFilter = queryInfo.getBsimFilter(); + + // Add threshold filters. + queryInfo.setMaximumResults(MAX_NUM_FUNCTIONS); + queryInfo.setSimilarityThreshold(SIMILARITY_BOUND); + queryInfo.setSignificanceThreshold(SIGNIFICANCE_BOUND); + + // Add prefilters. + setPrefilters(); + + // Add a simple date filter. + addBsimFilter(new DateLaterBSimFilterType(""), "01/01/1776"); + + // Demonstration of a filter that allows for multiple entries. All filters but the + // DateEarlier and DateLater allow this. The effect is that each filter will be OR'd + // with the others. This is effectively the same as creating three distinct ArchEquals filters. + // + // ie: "The architecture can equal x86:LE:64:default OR the architecture can equal + // ARM:LE_32:v4 OR ...." + addBsimFilter(new ArchitectureBSimFilterType(), + "x86:LE:64:default, x86:LE:32:default, ARM:LE:32:v4"); + + // Another filter with multiple entries, but in this case since it is a "NotEqual" filter, + // the items are "AND'd together. + // + // ie: "The compiler cannot equal windows AND the compiler cannot equal foo_compiler". + addBsimFilter(new CompilerBSimFilterType(), "windows, foo_compiler"); + + //connect to the database + try { + String dbUrl = + askString("", "Enter the URL of the BSim database:", "ghidra://localhost/bsimDb"); + queryService.initializeDatabase(dbUrl); + FunctionDatabase.Error error = queryService.getLastError(); + if (error != null && error.category == ErrorCategory.Nodatabase) { + println("Database [" + dbUrl + "] cannot be found (does it exist?)"); + return; + } + } + catch (QueryDatabaseException e) { + println(e.getMessage()); + return; + } + + // Execute query and print results. + List resultRows = executeQuery(queryInfo); + printFunctionQueryResults(resultRows, "\nFunction-level results before filtering"); + + // Add some simple post-query filters. These filters will only be applied to the result + // set returned from the previous query. + addBsimFilter(new Md5BSimFilterType(), currentProgram.getExecutableMD5()); + addBsimFilter(new CompilerBSimFilterType(), "gcc"); + addBsimFilter(new FunctionTagBSimFilterType("KNOWN_LIBRARY", queryService), + "false"); + + // Apply the filters and print results. + List filteredRows = + BSimMatchResult.filterMatchRows(bsimFilter, resultRows); + printFunctionQueryResults(filteredRows, "\nFunction-level results after filtering"); + printExecutableInformation(filteredRows); + } + + @Override + public void cleanup(boolean success) { + if (queryService != null) { + queryService.dispose(); + } + } + + /*********************************************************************** + * PRIVATE METHODS + ***********************************************************************/ + + /** + * Adds a filter to the given filter container. + * + * @param filterTemplate the filter type to add + * @param value the value of the filter + */ + private void addBsimFilter(BSimFilterType filterTemplate, String value) { + String[] inputs = value.split(","); + for (String input : inputs) { + if (!input.trim().isEmpty()) { + bsimFilter.addAtom(filterTemplate, input.trim()); + } + } + } + + /** + * Queries the database and returns the results. + * + * @param qInfo contains all information required for the query + * @return list of matches + * @throws QueryDatabaseException if there is a problem executing the query similar functions query + * @throws CancelledException if the user cancelled the operation + */ + private List executeQuery(SFQueryInfo qInfo) + throws QueryDatabaseException, CancelledException { + + SFQueryResult queryResults = queryService.querySimilarFunctions(qInfo, null, monitor); + List resultRows = + BSimMatchResult.generate(queryResults.getSimilarityResults(), currentProgram); + + return resultRows; + } + + /** + * Creates predicates that will be used to filter out functions. This example provides three + * different methods of doing this: + * + * - anonymous class + * - lambda + * - static method + * + * These are all possible because the filter takes a {@link BiPredicate}, which is a + * functional interface. + * + */ + private void setPrefilters() { + + PreFilter preFilter = queryInfo.getPreFilter(); + + // + // Option 1: Anonymous class + // Filters out any functions with a self significance less than a + // certain value. + // + preFilter.addPredicate(new BiPredicate() { + @Override + public boolean test(Program t, FunctionDescription u) { + return queryService.getLSHVectorFactory() + .getSelfSignificance( + u.getSignatureRecord().getLSHVector()) >= SELF_SIGNIFICANCE_BOUND; + } + }); + + // + // Option 2. Lambda expression + // Filters out any functions with a self significance less than a + // certain value. + // + preFilter.addPredicate((x, y) -> queryService.getLSHVectorFactory() + .getSelfSignificance( + y.getSignatureRecord().getLSHVector()) >= SELF_SIGNIFICANCE_BOUND); + + // + // Option 3. Static method + // Filters out any functions that are of type ANALYSIS. + // + preFilter.addPredicate(QueryWithFiltersScript::isNotAnalysisSourceType); + } + + /** + * Returns a set of ALL functions (no stubs) in the given program. + * + * @param program the program to get the functions from + * @return list of function symbols + */ + private HashSet getFunctionsToQuery(Program program) { + HashSet functions = new HashSet<>(); + FunctionIterator fIter = program.getFunctionManager().getFunctionsNoStubs(true); + for (Function func : fIter) { + functions.add((FunctionSymbol) func.getSymbol()); + } + return functions; + } + + /** + * Returns true if the given function is NOT an analysis type. + * + * @param program the current program + * @param funcDesc the function description object + * @return true if the symbol is NOT an analysis source type + */ + public static boolean isNotAnalysisSourceType(Program program, FunctionDescription funcDesc) { + Address address = + program.getAddressFactory().getDefaultAddressSpace().getAddress(funcDesc.getAddress()); + + Function function = program.getFunctionManager().getFunctionAt(address); + if (function == null || function.getName().equals(funcDesc.getFunctionName())) { + return false; + } + return function.getSymbol().getSource() != SourceType.ANALYSIS; + } + + /** + * Prints a sorted list of executables represented in the function matches. + * + * @param filteredRows list of function results + */ + private void printExecutableInformation(List filteredRows) { + + TreeSet execrows = ExecutableResult.generateFromMatchRows(filteredRows); + ExecutableResult[] results = new ExecutableResult[execrows.size()]; + results = execrows.toArray(results); + + Arrays.sort(results, new Comparator() { + @Override + public int compare(ExecutableResult o1, ExecutableResult o2) { + return Double.compare(o2.getSignificanceSum(), o1.getSignificanceSum()); + } + }); + + printf("Executable-level results:\n"); + for (int i = 0, max = Math.min(NUM_EXES_TO_DISPLAY, results.length); i < max; ++i) { + printf(" MD5: %s\n", results[i].getExecutableRecord().getMd5()); + printf(" Executable Name: %s\n", results[i].getExecutableRecord().getNameExec()); + printf(" Function Count: %d\n", results[i].getFunctionCount()); + printf(" Significance Sum: %f\n\n", results[i].getSignificanceSum()); + } + } + + /** + * Prints information about each function in the result set. + * + * @param resultRows the list of rows containing the info to print + * @param title the title to print + */ + private void printFunctionQueryResults(List resultRows, String title) { + printf(title + ": (%d)\n\n", resultRows.size()); + for (BSimMatchResult resultRow : resultRows) { + printf(" queried function: %s\n", + resultRow.getOriginalFunctionDescription().getFunctionName()); + printf(" matching function: %s\n", + resultRow.getMatchFunctionDescription().getFunctionName()); + printf(" executable of matching function: %s\n", + resultRow.getMatchFunctionDescription().getExecutableRecord().getNameExec()); + printf(" similarity: %f\n", resultRow.getSimilarity()); + printf(" significance: %f\n\n", resultRow.getSignificance()); + } + printf("\n"); + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/QueryWithFiltersScript.py b/Ghidra/Features/BSim/ghidra_scripts/QueryWithFiltersScript.py new file mode 100755 index 0000000000..00eaafe847 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/QueryWithFiltersScript.py @@ -0,0 +1,173 @@ +## ### +# IP: GHIDRA +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## +# Advanced example of BSim querying +# @category BSim.python + +import ghidra.query.facade.SimilarFunctionQueryService as SimilarFunctionQueryService +import ghidra.query.facade.SFQueryInfo as SFQueryInfo +import ghidra.query.FunctionDatabase as FunctionDatabase +import ghidra.query.facade.QueryDatabaseException as QueryDatabaseException +import java.util.HashSet as HashSet +import ghidra.app.plugin.core.query.QueryNearestRow as QueryNearestRow +import java.util.function.BiPredicate as BiPredicate +import ghidra.query.protocol.FilterTemplate as FilterTemplate +import ghidra.app.plugin.core.query.ExecutableResult as ExecutableResult +import java.util.Comparator as Comparator +import java.util.Arrays as Arrays +import java.lang.Double as Double + +#Query thresholds +MAX_NUM_FUNCTIONS = 100 +SIMILARITY_BOUND = 0.7 +SIGNIFICANCE_BOUND = 0.0 + +#limit the number of results displayed +NUM_EXES_TO_DISPLAY = 10 + +#for prefiltering: this number will be used to filter out small functions +SELF_SIGNIFICANCE_BOUND = 40.0 + +def run(): + + #get the set of functions to query + funcsToQuery = getFunctionsToQuery() + + #sets up the object required for querying the database + queryService = SimilarFunctionQueryService(currentProgram) + queryInfo = SFQueryInfo(funcsToQuery) + bsimFilter = queryInfo.getBsimFilter() + + #sets the query parameters. + #change the defined constants to control how fuzzy of + #a match you're willing to accept, and the maximum number + #of matches to return for each function + queryInfo.setMaximumResults(MAX_NUM_FUNCTIONS) + queryInfo.setSimilarityThreshold(SIMILARITY_BOUND) + queryInfo.setSignificanceThreshold(SIGNIFICANCE_BOUND) + + #add the prefilters + setPrefilters(queryService, queryInfo) + + #add a filter on the date + addBsimFilter(bsimFilter, FilterTemplate.DateLater(""), "01/01/1776") + + #add a filter with multiple values. Since this is an "Equal" filter, the results are OR'd together + #so a given executable will pass the main filter if it passes at least one of the subfilters + addBsimFilter(bsimFilter, FilterTemplate.ArchEquals(),"x86:LE:64:default, x86:LE:32:default, ARM:LE:32:v4") + + #now add a "notequal" filter + #to pass, the compiler can't be windows and it can't be foo_compiler + addBsimFilter(bsimFilter,FilterTemplate.CompNotEqual(),"windows, foo_compiler") + + #establish a connection to the BSim database + try: + dbUrl = askString("","Enter the URL of the BSim database:", "ghidra://localhost/bsimDB") + queryService.initializeDatabase(dbUrl) + error = queryService.getDatabase().getLastError() + if error is not None and (error.category is ErrorCategory.Nodatabase): + print "Database [%s] cannot be found (does it exist?)" % dbUrl + return + except QueryDatabaseException as e: + print e.getMessage() + return + + resultRows = executeQuery(queryService,queryInfo) + printFunctionQueryResults(resultRows, "\nFunction-level results before filtering") + + #now add some post-query filters, which filters the result set returned by the previous query + + addBsimFilter(bsimFilter, FilterTemplate.Md5NotEqual(), currentProgram.getExecutableMD5()) + addBsimFilter(bsimFilter, FilterTemplate.CompilerEquals(), "gcc") + addBsimFilter(bsimFilter, FilterTemplate.FunctionTagTemplate("KNOWN_LIBRARY", queryService), "false") + + #apply the filters and print the results + filteredRows = QueryNearestRow.filterMatchRows(bsimFilter, resultRows) + printFunctionQueryResults(filteredRows, "\nFunction-level results after filtering") + printExecutableInformation(filteredRows) + return + + +#collect the functions to query from currentProgram +def getFunctionsToQuery(): + functions = HashSet(); + fIter = currentProgram.getFunctionManager().getFunctionsNoStubs(True) + for func in fIter: + functions.add(func.getSymbol()) + return functions + +#query the database +def executeQuery(queryService,queryInfo): + queryResults = queryService.querySimilarFunctions(queryInfo,monitor) + resultRows = QueryNearestRow.generate(queryResults.getSimilarityResults(),currentProgram) + return resultRows + +def printFunctionQueryResults(resultRows, title): + print "%s: %d\n\n" % (title, resultRows.size()) + for row in resultRows: + print " queried function: %s" % row.getOriginalFunctionDescription().getFunctionName() + print " matching function: %s" % row.getMatchFunctionDescription().getFunctionName() + print " executable of matching function: %s" % row.getMatchFunctionDescription().getExecutableRecord().getNameExec() + print " similarity: %f" % row.getSimilarity() + print " significance: %f\n" % row.getSignificance() + +#Prefilters are used to filter out functions before sending a query to the database +#A typical use case would be to collect all functions in a binary, then use a +#prefilter to remove the functions with low self-significance (which is the +#"BSim way" to remove small functions) +def setPrefilters(queryService, queryInfo): + preFilter = queryInfo.getPreFilter(); + selfSigFilter = ExampleFilter(queryService) + preFilter.addPredicate(selfSigFilter) + +class ExampleFilter(BiPredicate): + + def __init__(self, queryService): + self.queryService = queryService + + def test(self,program, fdesc): + return self.queryService.getLSHVectorFactory().getSelfSignificance(fdesc.getSignatureRecord().getLSHVector()) >= SELF_SIGNIFICANCE_BOUND + +def addBsimFilter(bsimFilter, filterTemplate, values): + for value in values.split(","): + if len(value.strip()) > 0: + bsimFilter.addAtom(filterTemplate, value.strip(), FilterTemplate.Blank()) + +#calls the methods to aggregate executable-level information about the matches +def printExecutableInformation(filteredRows): + execrows = ExecutableResult.generateFromMatchRows(filteredRows) + results = execrows.toArray() + sorter = Sorter() + Arrays.sort(results,sorter) + print "Executable-level results:" + numExes = min(len(results),NUM_EXES_TO_DISPLAY) + for i in range (numExes): + print " MD5: %s" % results[i].getExecutableRecord().getMd5() + print " Executable Name: %s" % results[i].getExecutableRecord().getNameExec() + print " Function Count: %d" % results[i].getFunctionCount() + print " Significance Sum: %f\n" % results[i].getSignificanceSum() + return + +class Sorter(Comparator): + + def __init__(self): + return + + def compare(self,o1,o2): + return Double.compare(o2.getSignificanceSum(), o1.getSignificanceSum()) + + + +run() diff --git a/Ghidra/Features/BSim/ghidra_scripts/SetExecutableCategoryScript.java b/Ghidra/Features/BSim/ghidra_scripts/SetExecutableCategoryScript.java new file mode 100644 index 0000000000..7eea56de45 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/SetExecutableCategoryScript.java @@ -0,0 +1,45 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import org.apache.commons.lang3.StringUtils; + +import ghidra.app.script.GhidraScript; +import ghidra.framework.options.Options; +import ghidra.program.model.listing.Program; + +//@category BSim +//sets a property on the current program which can be used as +//an executable category in BSim +public class SetExecutableCategoryScript extends GhidraScript { + + @Override + protected void run() throws Exception { + if (currentProgram == null) { + popup("This script requires a program"); + return; + } + Options opts = currentProgram.getOptions(Program.PROGRAM_INFO); + String name = askString("Enter Property Name", "Name"); + if (StringUtils.isAllBlank(name)) { + return; + } + String value = askString("Enter Value of Property " + name, "Value"); + if (StringUtils.isAllBlank(value)) { + return; + } + opts.setString(name, value); + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/TailoredAnalysis.java b/Ghidra/Features/BSim/ghidra_scripts/TailoredAnalysis.java new file mode 100755 index 0000000000..06442edca5 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/TailoredAnalysis.java @@ -0,0 +1,56 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import ghidra.app.script.GhidraScript; +import ghidra.framework.options.Options; +import ghidra.program.model.listing.Program; + +// Setup tailored auto-analysis (in place of the headless analyzers full auto-analysis) +// suitable for BSim ingest process. Intended to be invoked as an analyzeHeadless -preScript +//@category BSim + +public class TailoredAnalysis extends GhidraScript { + + @Override + public void run() throws Exception { + Options pl = currentProgram.getOptions(Program.ANALYSIS_PROPERTIES); + pl.setBoolean("Decompiler Parameter ID", false); + + // These analyzers generate lots of cross references, which are not necessary for + // signature analysis, and take time to run. On the other hand, you may want + // them in general to facilitate general analysis + pl.setBoolean("Stack", false); +// pl.setBoolean("Windows x86 PE Instruction References", false); +// pl.setBoolean("Windows x86 PE C++", false); +// pl.setBoolean("Windows x86 PE Preliminary", false); +// pl.setBoolean("ELF Scalar Operand References", false); + + // Mangled symbols are good information but you may not be able to count on them being present in all versions +// Options analyzerOptions = pl.getOptions("Demangler"); +// analyzerOptions.setBoolean("Commit Function Signatures", false); + + // You really want these options turned on + pl.setBoolean("Shared Return Calls",true); + pl.setBoolean("Function Start Search", true); + pl.setBoolean("DWARF", false); +// Options analyzerOptions = pl.getOptions("Function Start Search"); +// analyzerOptions.setBoolean("Search Data Blocks", true); +// analyzerOptions = pl.getOptions("Function Start Search After Code"); +// analyzerOptions.setBoolean("Search Data Blocks", true); +// analyzerOptions = pl.getOptions("Function Start Search After Data"); +// analyzerOptions.setBoolean("Search Data Blocks", true); + } + +} diff --git a/Ghidra/Features/BSim/ghidra_scripts/UpdateBSimMetadata.java b/Ghidra/Features/BSim/ghidra_scripts/UpdateBSimMetadata.java new file mode 100755 index 0000000000..80962311e8 --- /dev/null +++ b/Ghidra/Features/BSim/ghidra_scripts/UpdateBSimMetadata.java @@ -0,0 +1,103 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +// Push updated information about function names and other metadata from the current program to a BSim database +//@category BSim + +import java.net.URL; + +import ghidra.app.script.GhidraScript; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.features.bsim.query.protocol.QueryUpdate; +import ghidra.features.bsim.query.protocol.ResponseUpdate; +import ghidra.program.model.listing.FunctionIterator; +import ghidra.program.model.listing.FunctionManager; + +public class UpdateBSimMetadata extends GhidraScript { + + @Override + protected void run() throws Exception { + if (currentProgram == null) { + return; + } + String bsim_url = System.getProperty("ghidra.bsimurl"); + if (bsim_url==null || bsim_url.length()==0) { + bsim_url = askString("Request Repository", "Select URL of database receiving update"); + } + + URL url = BSimClientFactory.deriveBSimURL(bsim_url); + try (FunctionDatabase database = BSimClientFactory.buildClient(url, true)) { + if (!database.initialize()) { + println(database.getLastError().message); + return; + } + println("Connected to " + database.getInfo().databasename); + + GenSignatures gensig = new GenSignatures(false); + gensig.setVectorFactory(database.getLSHVectorFactory()); + gensig.openProgram(currentProgram, null, null, null, null, null); + + FunctionManager functionManager = currentProgram.getFunctionManager(); + FunctionIterator funciter; + if (currentSelection != null) { + println("Scanning selected functions"); + funciter = functionManager.getFunctions(currentSelection, true); + } + else { + println("Scanning all functions"); + funciter = functionManager.getFunctions(true); // If no highlight, update all functions + } + gensig.scanFunctionsMetadata(funciter, monitor); + QueryUpdate update = new QueryUpdate(); + update.manage = gensig.getDescriptionManager(); + + ResponseUpdate respup = update.execute(database); // Try to push the update + if (respup == null) { + println(database.getLastError().message); + return; + } + if (!respup.badexe.isEmpty()) { + for (int j = 0; j < respup.badexe.size(); ++j) { + ExecutableRecord erec = respup.badexe.get(j); + println("Database does not contain executable: " + erec.getNameExec()); + } + } + if (!respup.badfunc.isEmpty()) { + int max = respup.badfunc.size(); + if (max > 10) { + println( + "Could not find " + Integer.toString(respup.badfunc.size()) + " functions"); + max = 10; + } + for (int j = 0; j < max; ++j) { + FunctionDescription func = respup.badfunc.get(j); + println("Could not update function " + func.getFunctionName()); + } + } + if (respup.exeupdate > 0) { + println("Updated executable metadata"); + } + if (respup.funcupdate > 0) { + println("Updated " + Integer.toString(respup.funcupdate) + " functions"); + } + if (respup.exeupdate == 0 && respup.funcupdate == 0) { + println("No changes"); + } + } + } + +} diff --git a/Ghidra/Features/BSim/make-postgres.sh b/Ghidra/Features/BSim/make-postgres.sh new file mode 100755 index 0000000000..b06d235288 --- /dev/null +++ b/Ghidra/Features/BSim/make-postgres.sh @@ -0,0 +1,126 @@ +#!/bin/bash +## ### +# IP: GHIDRA +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +## +# +# This script may be used to build the postgresql server within +# a GHIDRA installation. The postgresql server configuration options +# below (POSTGRES_CONFIG_OPTIONS) may be adjusted if required +# (e.g., build without openssl use, etc.). +# +# See https://www.postgresql.org/docs/10/install-procedure.html +# for supported postgresql config options. +# +# Additional packages may need to be installed include to perform the +# postgresql build. Please refer to the following web page for +# package dependencies: +# +# https://wiki.postgresql.org/wiki/Compile_and_Install_from_source_code +# +# The postgresql source distribution should reside within the BSim module +# directory prior to running this script. Within development environments +# it will first check the ghidra.bin repo for this source file. +# + +POSTGRES=postgresql-15.3 +POSTGRES_GZ=${POSTGRES}.tar.gz +POSTGRES_CONFIG_OPTIONS="--disable-rpath --with-openssl" + +DIR=$(cd `dirname $0`; pwd) + +POSTGRES_GZ_PATH=${DIR}/../../../../ghidra.bin/Ghidra/Features/BSim/${POSTGRES_GZ} +if [ ! -f "${POSTGRES_GZ_PATH}" ]; then + POSTGRES_GZ_PATH=${DIR}/${POSTGRES_GZ} + if [ ! -f "${POSTGRES_GZ_PATH}" ]; then + echo "Postgres source bundle not found: ${POSTGRES_GZ_PATH}" + exit -1 + fi +fi + +OS=`uname -s` +ARCH=`arch` + +cd ${DIR} + +mkdir -p build > /dev/null + +if [ ! -d build/${POSTGRES} ]; then + # Unpack postgres source distro into build + echo "Unpacking postgresql source: ${POSTGRES_GZ_PATH}" + $(cd build; tar -xzf ${POSTGRES_GZ_PATH} ) +fi + +# Build postgresql + +pushd build/${POSTGRES} + +if [ "$OS" = "Darwin" ]; then + export MACOSX_DEPLOYMENT_TARGET=10.5 + export ARCHFLAGS="-arch x86_64" + OSDIR=mac_x86_64 +elif [ "$ARCH" = "x86_64" ]; then + OSDIR=linux_x86_64 +else + echo "Unsupported platform: $OS $ARCH" + exit -1 +fi + +# Install within build/os +INSTALL_DIR=${DIR}/build/os/${OSDIR}/postgresql +rm -rf ${INSTALL_DIR} > /dev/null + +make distclean + +# Configure postgres + +./configure ${POSTGRES_CONFIG_OPTIONS} --prefix=${INSTALL_DIR} +if [ $? != 0 ]; then + exit $? +fi + +make install +if [ $? != 0 ]; then + exit $? +fi + +make -C contrib/pg_prewarm install +if [ $? != 0 ]; then + exit $? +fi + +echo "Completed postgresql build" + +# Build lshvector plugin for postgresql + +popd + +rm -rf build/lshvector > /dev/null +mkdir build/lshvector + +echo "Building lshvector plugin..." + +cp src/lshvector/* build/lshvector +cp src/lshvector/c/* build/lshvector + +cd build/lshvector +make -f Makefile.lshvector install PG_CONFIG=${INSTALL_DIR}/bin/pg_config + +if [ $? = 0 ]; then + echo "Completed build and install of lshvector postgresql plugin" + exit 0 +fi + +exit -1 + diff --git a/Ghidra/Features/BSim/other/testscripts/InstallMetadataTest.java b/Ghidra/Features/BSim/other/testscripts/InstallMetadataTest.java new file mode 100755 index 0000000000..b7918fb670 --- /dev/null +++ b/Ghidra/Features/BSim/other/testscripts/InstallMetadataTest.java @@ -0,0 +1,34 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import ghidra.app.script.GhidraScript; +import ghidra.framework.options.Options; +import ghidra.program.model.listing.Program; + +/** + * This script is used by the unit test BSimServerTest + */ +public class InstallMetadataTest extends GhidraScript { + + @Override + protected void run() throws Exception { + Options pl = currentProgram.getOptions(Program.PROGRAM_INFO); + String value = "static"; + if (currentProgram.getName().contains(".so")) + value = "shared"; + pl.setString("Test Category", value); + } + +} diff --git a/Ghidra/Features/BSim/other/testscripts/RegressionSignatures.java b/Ghidra/Features/BSim/other/testscripts/RegressionSignatures.java new file mode 100755 index 0000000000..a261d67b10 --- /dev/null +++ b/Ghidra/Features/BSim/other/testscripts/RegressionSignatures.java @@ -0,0 +1,69 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +import java.io.File; +import java.io.FileWriter; +import java.io.IOException; +import java.util.ArrayList; +import java.util.Iterator; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.script.GhidraScript; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.FunctionManager; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.GenSignatures; +import ghidra.features.bsim.query.client.Configuration; +import ghidra.features.bsim.query.description.DescriptionManager; + +/** + * This script is used by the unit test BSimServerTest + */ +public class RegressionSignatures extends GhidraScript { + + @Override + protected void run() throws Exception { + String md5string = currentProgram.getExecutableMD5(); + if ((md5string == null) || (md5string.length() < 10)) + throw new IOException("Could not get MD5 on file: " + currentProgram.getName()); + String basename = "sigs_" + md5string; + File file = null; + // This form of askString will work for both standalone execution or for parallel + File workingdir = askDirectory("RegressionSignatures:", "Working directory"); + file = new File(workingdir, basename); + + LSHVectorFactory vectorFactory = FunctionDatabase.generateLSHVectorFactory(); + Configuration config = FunctionDatabase.loadConfigurationTemplate("medium_64"); + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + GenSignatures gensig = new GenSignatures(true); + gensig.setVectorFactory(vectorFactory); + + List names = new ArrayList(); + names.add("Test Category"); + gensig.addExecutableCategories(names); + String repo = "ghidra://localhost/repo"; + String path = "/raw"; + gensig.openProgram(this.currentProgram, null, null, null, repo, path); + FunctionManager fman = currentProgram.getFunctionManager(); + Iterator iter = fman.getFunctions(true); + gensig.scanFunctions(iter, fman.getFunctionCount(), monitor); + FileWriter fwrite = new FileWriter(file); + DescriptionManager manager = gensig.getDescriptionManager(); + manager.saveXml(fwrite); + fwrite.close(); + } + +} diff --git a/Ghidra/Features/BSim/src/lshvector/Makefile.lshvector b/Ghidra/Features/BSim/src/lshvector/Makefile.lshvector new file mode 100755 index 0000000000..13069f6335 --- /dev/null +++ b/Ghidra/Features/BSim/src/lshvector/Makefile.lshvector @@ -0,0 +1,25 @@ +# Locality Sensitive Hashing package +# NOTE: This file cannot be executed in place. It is copied into a temporary +# directory with its source code and executed there. + +ifeq ($(PG_CONFIG),) +default: + echo "You must specifiy PG_CONFIG" + false + +endif + +MODULE_big = lshvector +OBJS= lsh.o weights.o binhash.o crc32.o + +EXTENSION = lshvector +DATA = lshvector--1.0.sql + +REGRESS = lshvector + +EXTRA_CLEAN = + +SHLIB_LINK += $(filter -lm, $(LIBS)) + +PGXS := $(shell $(PG_CONFIG) --pgxs) +include $(PGXS) diff --git a/Ghidra/Features/BSim/src/lshvector/c/binhash.c b/Ghidra/Features/BSim/src/lshvector/c/binhash.c new file mode 100755 index 0000000000..8dba340305 --- /dev/null +++ b/Ghidra/Features/BSim/src/lshvector/c/binhash.c @@ -0,0 +1,277 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "lsh.h" + +#define LSH_HASHBASE 0xD7E6A299 + +static char hash_signtable[512]; + +static void hash_int_fft_16(int32 *arr) + +{ + int32 x,y; + + x = arr[0]; y = arr[8]; arr[0] = x + y; arr[8] = x - y; + x = arr[1]; y = arr[9]; arr[1] = x + y; arr[9] = x - y; + x = arr[2]; y = arr[10]; arr[2] = x + y; arr[10] = x - y; + x = arr[3]; y = arr[11]; arr[3] = x + y; arr[11] = x - y; + x = arr[4]; y = arr[12]; arr[4] = x + y; arr[12] = x - y; + x = arr[5]; y = arr[13]; arr[5] = x + y; arr[13] = x - y; + x = arr[6]; y = arr[14]; arr[6] = x + y; arr[14] = x - y; + x = arr[7]; y = arr[15]; arr[7] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[4]; arr[0] = x + y; arr[4] = x - y; + x = arr[1]; y = arr[5]; arr[1] = x + y; arr[5] = x - y; + x = arr[2]; y = arr[6]; arr[2] = x + y; arr[6] = x - y; + x = arr[3]; y = arr[7]; arr[3] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[12]; arr[8] = x + y; arr[12] = x - y; + x = arr[9]; y = arr[13]; arr[9] = x + y; arr[13] = x - y; + x = arr[10]; y = arr[14]; arr[10] = x + y; arr[14] = x - y; + x = arr[11]; y = arr[15]; arr[11] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[2]; arr[0] = x + y; arr[2] = x - y; + x = arr[1]; y = arr[3]; arr[1] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[6]; arr[4] = x + y; arr[6] = x - y; + x = arr[5]; y = arr[7]; arr[5] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[10]; arr[8] = x + y; arr[10] = x - y; + x = arr[9]; y = arr[11]; arr[9] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[14]; arr[12] = x + y; arr[14] = x - y; + x = arr[13]; y = arr[15]; arr[13] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[1]; arr[0] = x + y; arr[1] = x - y; + x = arr[2]; y = arr[3]; arr[2] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[5]; arr[4] = x + y; arr[5] = x - y; + x = arr[6]; y = arr[7]; arr[6] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[9]; arr[8] = x + y; arr[9] = x - y; + x = arr[10]; y = arr[11]; arr[10] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[13]; arr[12] = x + y; arr[13] = x - y; + x = arr[14]; y = arr[15]; arr[14] = x + y; arr[15] = x - y; +} + +static void hash_double_fft_16(double *arr) + +{ + double x,y; + + x = arr[0]; y = arr[8]; arr[0] = x + y; arr[8] = x - y; + x = arr[1]; y = arr[9]; arr[1] = x + y; arr[9] = x - y; + x = arr[2]; y = arr[10]; arr[2] = x + y; arr[10] = x - y; + x = arr[3]; y = arr[11]; arr[3] = x + y; arr[11] = x - y; + x = arr[4]; y = arr[12]; arr[4] = x + y; arr[12] = x - y; + x = arr[5]; y = arr[13]; arr[5] = x + y; arr[13] = x - y; + x = arr[6]; y = arr[14]; arr[6] = x + y; arr[14] = x - y; + x = arr[7]; y = arr[15]; arr[7] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[4]; arr[0] = x + y; arr[4] = x - y; + x = arr[1]; y = arr[5]; arr[1] = x + y; arr[5] = x - y; + x = arr[2]; y = arr[6]; arr[2] = x + y; arr[6] = x - y; + x = arr[3]; y = arr[7]; arr[3] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[12]; arr[8] = x + y; arr[12] = x - y; + x = arr[9]; y = arr[13]; arr[9] = x + y; arr[13] = x - y; + x = arr[10]; y = arr[14]; arr[10] = x + y; arr[14] = x - y; + x = arr[11]; y = arr[15]; arr[11] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[2]; arr[0] = x + y; arr[2] = x - y; + x = arr[1]; y = arr[3]; arr[1] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[6]; arr[4] = x + y; arr[6] = x - y; + x = arr[5]; y = arr[7]; arr[5] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[10]; arr[8] = x + y; arr[10] = x - y; + x = arr[9]; y = arr[11]; arr[9] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[14]; arr[12] = x + y; arr[14] = x - y; + x = arr[13]; y = arr[15]; arr[13] = x + y; arr[15] = x - y; + + x = arr[0]; y = arr[1]; arr[0] = x + y; arr[1] = x - y; + x = arr[2]; y = arr[3]; arr[2] = x + y; arr[3] = x - y; + x = arr[4]; y = arr[5]; arr[4] = x + y; arr[5] = x - y; + x = arr[6]; y = arr[7]; arr[6] = x + y; arr[7] = x - y; + x = arr[8]; y = arr[9]; arr[8] = x + y; arr[9] = x - y; + x = arr[10]; y = arr[11]; arr[10] = x + y; arr[11] = x - y; + x = arr[12]; y = arr[13]; arr[12] = x + y; arr[13] = x - y; + x = arr[14]; y = arr[15]; arr[14] = x + y; arr[15] = x - y; +} + +/* + * This is a precalculated table for generating dotproducts with the random family of vectors directly + * The first vector r_0 is expressed as a hashing function on the dimension index and the other vectors + * are derived from r_0 using an FFT. The table is formed by precalculating the FFT on basis vectors in this table + */ +void lsh_setup_signtable(void) + +{ + int32 i,j; + int32 arr[16]; + char *hibit0ptr; + char *hibit1ptr; + + for(i=0;i<16;++i) { /* For each 4-bit position */ + hibit0ptr = hash_signtable + i * 16; + hibit1ptr = hash_signtable + (i+16) * 16; + for(j=0;j<16;++j) + arr[j] = 0; + + arr[ i ] = 1; + hash_int_fft_16(arr); + for(j=0;j<16;++j) { + if (arr[j] > 0) { + hibit0ptr[j] = '+'; + hibit1ptr[j] = '-'; + } + else { + hibit0ptr[j] = '-'; + hibit1ptr[j] = '+'; + } + } + } +} + +/* + * Generate a dot product of the hash vector in -vec- with a random family of 16 vectors, { r } + * r_0 is a randomly generated set of +1 -1 coefficients across all the dimensions (indexed by uint32 vec[i].hash) + * The coefficient is calculated as a hashing function from the seed -hashcur- and the index (vec[i].hash), + * so it should be balanced between +1 and -1. + * All the other vectors are generated from an FFT of r_0. This allows the dotproduct with vec to be calculated + * using an FFT if -vec- has many non-zero coefficients. If -vec- has only a few non-zero coefficients, + * the dotproduct if calculated with each vector in the family directly for better efficiency. + * The resulting dotproducts are converted into a 16-long bitvector based on the sign of the dotproduct and + * placed in -bucket- + */ +static uint32 hash_16_dotproduct(uint32 bucket,LSH_ITEM *vec,uint32 vecsize,uint32 hashcur,uint32 vecsizeupper) + +{ + uint32 i,j; + uint32 rownum; + char *signptr; + double res[16]; + + for(i=0;i<16;++i) + res[i] = 0.0; /* Initialize the dotproduct results to zero */ + + if (vecsize < vecsizeupper) { /* If there are a small number of non-zero coefficients in -vec- */ + for(i=0;i>24)&0x1f; + signptr = hash_signtable + rownum * 16; + for(j=0;j<16;++j) { /* Based on the precalculated coeff table calculate this portion of dotproduct */ + if (signptr[j] == '+') + res[j] += vec[i].coeff; /* Dot product with +1 coeff */ + else + res[j] -= vec[i].coeff; /* Dot product with -1 coeff */ + } + } + } + else { /* If we have many non-zero coeffs in -vec- */ + for(i=0;i>24)&0x1f; + if (rownum < 0x10) /* Set-up for the FFT */ + res[rownum] += vec[i].coeff; + else + res[rownum&0xf] -= vec[i].coeff; + } + hash_double_fft_16(res); /* Calculate the remaining dotproducts be performing FFT */ + } + + for(i=0;i<16;++i) { /* Convert the dotproduct results to a bitvector */ + bucket <<= 1; + if (res[i] > 0.0) + bucket |= 1; + } + return bucket; +} + +void lsh_generate_binids(uint32 *res,LSH_ITEM *vec,uint32 vecsize) + +{ + uint32 bucket = 0; + int32 bucketcnt = 0; + int32 i,bitsleft; + uint32 curid; + uint32 mask,val; + uint32 hashbase = LSH_HASHBASE; + + for(i=0;i= bitsleft) { + curid <<= bitsleft; + mask = 1; + mask = (mask << bitsleft)-1; + val = bucket >> (bucketcnt - bitsleft); + curid |= (val & mask); + bucketcnt -= bitsleft; + bitsleft = 0; + } + else { + curid <<= bucketcnt; + mask = 1; + mask = (mask << bucketcnt)-1; + curid |= (bucket & mask); + bitsleft -= bucketcnt; + bucketcnt = 0; + } + } while(bitsleft > 0); + res[ i ] = curid; + } +} + +void lsh_generate_binids_datum(Datum *res,LSH_ITEM *vec,uint32 vecsize) + +{ + uint32 bucket = 0; + int32 bucketcnt = 0; + int32 i,bitsleft; + uint32 curid; + uint32 mask,val; + uint32 hashbase = LSH_HASHBASE; + + for(i=0;i= bitsleft) { + curid <<= bitsleft; + mask = 1; + mask = (mask << bitsleft)-1; + val = bucket >> (bucketcnt - bitsleft); + curid |= (val & mask); + bucketcnt -= bitsleft; + bitsleft = 0; + } + else { + curid <<= bucketcnt; + mask = 1; + mask = (mask << bucketcnt)-1; + curid |= (bucket & mask); + bitsleft -= bucketcnt; + bucketcnt = 0; + } + } while(bitsleft > 0); + res[ i ] = Int32GetDatum((int32)curid); + } +} diff --git a/Ghidra/Features/BSim/src/lshvector/c/crc32.c b/Ghidra/Features/BSim/src/lshvector/c/crc32.c new file mode 100755 index 0000000000..cd9b717269 --- /dev/null +++ b/Ghidra/Features/BSim/src/lshvector/c/crc32.c @@ -0,0 +1,101 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "lsh.h" + +#define CRC_UPDATE(REG,VAL) (crc32tab[ (REG ^ VAL)&0xff ] ^ (REG >> 8)) + +/* Table for bytewise calculation of a 32-bit Cyclic Redundancy Check */ +uint32 crc32tab[] = { + 0x0,0x77073096,0xee0e612c,0x990951ba,0x76dc419,0x706af48f, + 0xe963a535,0x9e6495a3,0xedb8832,0x79dcb8a4,0xe0d5e91e, + 0x97d2d988,0x9b64c2b,0x7eb17cbd,0xe7b82d07,0x90bf1d91, + 0x1db71064,0x6ab020f2,0xf3b97148,0x84be41de,0x1adad47d, + 0x6ddde4eb,0xf4d4b551,0x83d385c7,0x136c9856,0x646ba8c0, + 0xfd62f97a,0x8a65c9ec,0x14015c4f,0x63066cd9,0xfa0f3d63, + 0x8d080df5,0x3b6e20c8,0x4c69105e,0xd56041e4,0xa2677172, + 0x3c03e4d1,0x4b04d447,0xd20d85fd,0xa50ab56b,0x35b5a8fa, + 0x42b2986c,0xdbbbc9d6,0xacbcf940,0x32d86ce3,0x45df5c75, + 0xdcd60dcf,0xabd13d59,0x26d930ac,0x51de003a,0xc8d75180, + 0xbfd06116,0x21b4f4b5,0x56b3c423,0xcfba9599,0xb8bda50f, + 0x2802b89e,0x5f058808,0xc60cd9b2,0xb10be924,0x2f6f7c87, + 0x58684c11,0xc1611dab,0xb6662d3d,0x76dc4190,0x1db7106, + 0x98d220bc,0xefd5102a,0x71b18589,0x6b6b51f,0x9fbfe4a5, + 0xe8b8d433,0x7807c9a2,0xf00f934,0x9609a88e,0xe10e9818, + 0x7f6a0dbb,0x86d3d2d,0x91646c97,0xe6635c01,0x6b6b51f4, + 0x1c6c6162,0x856530d8,0xf262004e,0x6c0695ed,0x1b01a57b, + 0x8208f4c1,0xf50fc457,0x65b0d9c6,0x12b7e950,0x8bbeb8ea, + 0xfcb9887c,0x62dd1ddf,0x15da2d49,0x8cd37cf3,0xfbd44c65, + 0x4db26158,0x3ab551ce,0xa3bc0074,0xd4bb30e2,0x4adfa541, + 0x3dd895d7,0xa4d1c46d,0xd3d6f4fb,0x4369e96a,0x346ed9fc, + 0xad678846,0xda60b8d0,0x44042d73,0x33031de5,0xaa0a4c5f, + 0xdd0d7cc9,0x5005713c,0x270241aa,0xbe0b1010,0xc90c2086, + 0x5768b525,0x206f85b3,0xb966d409,0xce61e49f,0x5edef90e, + 0x29d9c998,0xb0d09822,0xc7d7a8b4,0x59b33d17,0x2eb40d81, + 0xb7bd5c3b,0xc0ba6cad,0xedb88320,0x9abfb3b6,0x3b6e20c, + 0x74b1d29a,0xead54739,0x9dd277af,0x4db2615,0x73dc1683, + 0xe3630b12,0x94643b84,0xd6d6a3e,0x7a6a5aa8,0xe40ecf0b, + 0x9309ff9d,0xa00ae27,0x7d079eb1,0xf00f9344,0x8708a3d2, + 0x1e01f268,0x6906c2fe,0xf762575d,0x806567cb,0x196c3671, + 0x6e6b06e7,0xfed41b76,0x89d32be0,0x10da7a5a,0x67dd4acc, + 0xf9b9df6f,0x8ebeeff9,0x17b7be43,0x60b08ed5,0xd6d6a3e8, + 0xa1d1937e,0x38d8c2c4,0x4fdff252,0xd1bb67f1,0xa6bc5767, + 0x3fb506dd,0x48b2364b,0xd80d2bda,0xaf0a1b4c,0x36034af6, + 0x41047a60,0xdf60efc3,0xa867df55,0x316e8eef,0x4669be79, + 0xcb61b38c,0xbc66831a,0x256fd2a0,0x5268e236,0xcc0c7795, + 0xbb0b4703,0x220216b9,0x5505262f,0xc5ba3bbe,0xb2bd0b28, + 0x2bb45a92,0x5cb36a04,0xc2d7ffa7,0xb5d0cf31,0x2cd99e8b, + 0x5bdeae1d,0x9b64c2b0,0xec63f226,0x756aa39c,0x26d930a, + 0x9c0906a9,0xeb0e363f,0x72076785,0x5005713,0x95bf4a82, + 0xe2b87a14,0x7bb12bae,0xcb61b38,0x92d28e9b,0xe5d5be0d, + 0x7cdcefb7,0xbdbdf21,0x86d3d2d4,0xf1d4e242,0x68ddb3f8, + 0x1fda836e,0x81be16cd,0xf6b9265b,0x6fb077e1,0x18b74777, + 0x88085ae6,0xff0f6a70,0x66063bca,0x11010b5c,0x8f659eff, + 0xf862ae69,0x616bffd3,0x166ccf45,0xa00ae278,0xd70dd2ee, + 0x4e048354,0x3903b3c2,0xa7672661,0xd06016f7,0x4969474d, + 0x3e6e77db,0xaed16a4a,0xd9d65adc,0x40df0b66,0x37d83bf0, + 0xa9bcae53,0xdebb9ec5,0x47b2cf7f,0x30b5ffe9,0xbdbdf21c, + 0xcabac28a,0x53b39330,0x24b4a3a6,0xbad03605,0xcdd70693, + 0x54de5729,0x23d967bf,0xb3667a2e,0xc4614ab8,0x5d681b02, + 0x2a6f2b94,0xb40bbe37,0xc30c8ea1,0x5a05df1b,0x2d02ef8d }; + +uint64 lsh_hash_internal(LSHVECTOR *vec) + +{ + uint32 reg1,reg2; + uint32 curtf,curhash,oldreg1; + uint32 i; + uint64 res; + + reg1 = 0x12CF93AB; + reg2 = 0xEE39B2D6; + + for(i=0;inumitems;++i) { + curtf = vec->items[i].tf; + curhash = vec->items[i].hash; + oldreg1 = reg1; + reg1 = CRC_UPDATE(reg1,curtf); + reg1 = CRC_UPDATE(reg1,curhash); + reg1 = CRC_UPDATE(reg1,(reg2>>24)); + reg2 = CRC_UPDATE(reg2,(oldreg1>>24)); + reg2 = CRC_UPDATE(reg2,(curhash>>8)); + reg2 = CRC_UPDATE(reg2,(curhash>>16)); + reg2 = CRC_UPDATE(reg2,(curhash>>24)); + } + res = reg1; + res <<= 32; + res |= reg2; + return res; +} diff --git a/Ghidra/Features/BSim/src/lshvector/c/lsh.c b/Ghidra/Features/BSim/src/lshvector/c/lsh.c new file mode 100755 index 0000000000..ac362f66a7 --- /dev/null +++ b/Ghidra/Features/BSim/src/lshvector/c/lsh.c @@ -0,0 +1,414 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "lsh.h" +#include "fmgr.h" +#include "funcapi.h" +#include "access/htup_details.h" +#include "access/gin.h" +#include "libpq/pqformat.h" +#include + +PG_MODULE_MAGIC; + +void _PG_init(void); + +PG_FUNCTION_INFO_V1(lshvector_in); +PG_FUNCTION_INFO_V1(lshvector_out); +PG_FUNCTION_INFO_V1(lshvector_send); +PG_FUNCTION_INFO_V1(lshvector_recv); +PG_FUNCTION_INFO_V1(lshvector_hash); +PG_FUNCTION_INFO_V1(lshvector_compare); +PG_FUNCTION_INFO_V1(lshvector_overlap); + +PG_FUNCTION_INFO_V1(lshvector_gin_extract_value); +PG_FUNCTION_INFO_V1(lshvector_gin_extract_query); +PG_FUNCTION_INFO_V1(lshvector_gin_consistent); + +PG_FUNCTION_INFO_V1(lsh_load); +PG_FUNCTION_INFO_V1(lsh_reload); +PG_FUNCTION_INFO_V1(lsh_getweight); + +Datum lshvector_in(PG_FUNCTION_ARGS); +Datum lshvector_out(PG_FUNCTION_ARGS); +Datum lshvector_send(PG_FUNCTION_ARGS); +Datum lshvector_recv(PG_FUNCTION_ARGS); +Datum lshvector_hash(PG_FUNCTION_ARGS); +Datum lshvector_compare(PG_FUNCTION_ARGS); +Datum lshvector_overlap(PG_FUNCTION_ARGS); + +Datum lshvector_gin_extract_value(PG_FUNCTION_ARGS); +Datum lshvector_gin_extract_query(PG_FUNCTION_ARGS); +Datum lshvector_gin_consistent(PG_FUNCTION_ARGS); + +Datum lsh_load(PG_FUNCTION_ARGS); +Datum lsh_reload(PG_FUNCTION_ARGS); +Datum lsh_getweight(PG_FUNCTION_ARGS); + +/* + * Allocate memory for an LSHVECTOR given the raw count of the number of hash entries in the vector + */ +static LSHVECTOR *allocate_lshvector(uint32 numentries) + +{ + LSHVECTOR *out; + uint32 maxitems, commonlen; + + /* Maximum number of hashes in a single LSHVECTOR assuming a 1 gigabyte allocation limit */ + maxitems = (0x3fffffff - HDRSIZELSH) / sizeof(LSH_ITEM); + + if (numentries > maxitems) { + ereport(ERROR,(errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),errmsg("Exceeded maximum entries for single lshvector"))); + /* Does not return */ + } + commonlen = HDRSIZELSH + numentries * sizeof(LSH_ITEM); + out = (LSHVECTOR *) palloc(commonlen); + SET_VARSIZE(out,commonlen); + return out; +} + +void _PG_init(void) + +{ + lsh_initialize(); +} + +Datum lsh_load(PG_FUNCTION_ARGS) + +{ + if (!weights_loaded) { + lsh_load_weights(); + lsh_load_lookuptable(); + lsh_load_binconfig(); + weights_loaded = true; + } + PG_RETURN_INT32(0); +} + +Datum lsh_reload(PG_FUNCTION_ARGS) + +{ + lsh_load_weights(); + lsh_load_lookuptable(); + lsh_load_binconfig(); + weights_loaded = true; + PG_RETURN_INT32(0); +} + +Datum lsh_getweight(PG_FUNCTION_ARGS) + +{ + LSHVECTOR *vec = PG_GETARG_LSHVECTOR_P(0); + uint32 arg = PG_GETARG_UINT32(1); + double res; + + if (arg >= vec->numitems) + res = 0.0; + else + res = vec->items[arg].coeff; + PG_FREE_IF_COPY(vec,0); + PG_RETURN_FLOAT8( res ); +} + +/* + * text input + */ +Datum +lshvector_in(PG_FUNCTION_ARGS) +{ + char *buf = (char *) PG_GETARG_POINTER(0); + char *ptr,*ptrstart; + LSHVECTOR *vec; + uint32 numitems = 0; + uint32 commacount = 0; + uint32 i,j; + int32 val; + char curc; + + ptr = buf; + curc = '\0'; + while(*ptr) { + curc = *ptr; + if (isspace(curc)==0) break; + ++ptr; + } + if (curc != '(') + ereport(ERROR,(errcode(ERRCODE_SYNTAX_ERROR),errmsg("Missing opening '('"))); /* Does not return */ + ++ptr; + ptrstart = ptr; + while (*ptr) { + curc = *ptr; + if (curc == ':') + numitems += 1; + else if (curc == ',') + commacount += 1; + else if (curc == ')') + break; + ++ptr; + } + if ((curc != ')')||(numitems != commacount+1)) + ereport(ERROR,(errcode(ERRCODE_SYNTAX_ERROR),errmsg("Bad delimiters"))); /* Does not return */ + + vec = allocate_lshvector(numitems); + + ptr = ptrstart; + i = 0; + j = 0; + while(*ptr) { + val = strtol(ptr,&ptr,16); + if (j==0) { + if ((val<1)||(val>64)) { + pfree(vec); + ereport(ERROR,(errcode(ERRCODE_SYNTAX_ERROR),errmsg("Term frequency count out of bounds"))); /* Does not return */ + } + vec->items[i].tf = (uint16)val; + j = 1; + } + else { + vec->items[i].hash = (uint32)val; + vec->items[i].idf = 0; + j = 0; + i += 1; + } + while(isspace( *ptr )) + ptr++; + if (*ptr == ')') break; + if (*ptr == ':') { + if (j==0) { + pfree(vec); + ereport(ERROR,(errcode(ERRCODE_SYNTAX_ERROR),errmsg("Expected ','"))); /* Does not return */ + } + ptr++; + } + else if (*ptr == ',') { + if (j==1) { + pfree(vec); + ereport(ERROR,(errcode(ERRCODE_SYNTAX_ERROR),errmsg("Expected ':'"))); /* Does not return */ + } + ptr++; + } + } + vec->numitems = numitems; + lsh_calc_weights(vec); + PG_RETURN_POINTER(vec); +} + +/* + * text output + */ +Datum +lshvector_out(PG_FUNCTION_ARGS) +{ + LSHVECTOR *vec = PG_GETARG_LSHVECTOR_P(0); + StringInfoData buf; + uint32 i,sz; + + initStringInfo(&buf); + + appendStringInfoChar(&buf,'('); + sz = vec->numitems; + for(i=0;iitems[i].tf); + appendStringInfoChar(&buf,':'); + appendStringInfo(&buf,"%x",(int32)vec->items[i].hash); + if (i+1 < sz) + appendStringInfoChar(&buf,','); + } + appendStringInfoChar(&buf,')'); + + PG_FREE_IF_COPY(vec,0); + + PG_RETURN_CSTRING(buf.data); +} + +/* + * binary output + */ +Datum +lshvector_send(PG_FUNCTION_ARGS) +{ + LSHVECTOR *vec = PG_GETARG_LSHVECTOR_P(0); + uint32 i; + uint32 numitems; + StringInfoData buf; + + numitems = vec->numitems; + + pq_begintypsend(&buf); + pq_sendint(&buf,numitems,4); + + for(i=0;iitems[i].tf,1); + pq_sendint(&buf,vec->items[i].hash,4); + } + PG_FREE_IF_COPY(vec,0); + PG_RETURN_BYTEA_P(pq_endtypsend(&buf)); +} + +/* + * binary input + */ +Datum +lshvector_recv(PG_FUNCTION_ARGS) +{ + LSHVECTOR *out; + StringInfo buf = (StringInfo) PG_GETARG_POINTER(0); + uint32 numitems; + uint32 tf; + uint32 i; + + numitems = pq_getmsgint(buf,4); + out = allocate_lshvector(numitems); + + out->numitems = numitems; + for(i=0;i64)) { + pfree(out); + ereport(ERROR,(errcode(ERRCODE_NUMERIC_VALUE_OUT_OF_RANGE),errmsg("Term frequency is out of range"))); + /* Does not return */ + } + out->items[i].tf = tf; + out->items[i].hash = pq_getmsgint(buf,4); + } + lsh_calc_weights(out); + PG_RETURN_POINTER(out); +} + +Datum lshvector_hash(PG_FUNCTION_ARGS) +{ + LSHVECTOR *a = PG_GETARG_LSHVECTOR_P(0); + int64 res = (int64)lsh_hash_internal(a); + + PG_FREE_IF_COPY(a,0); + + PG_RETURN_INT64(res); +} + +Datum lshvector_compare(PG_FUNCTION_ARGS) +{ + LSHVECTOR *a = PG_GETARG_LSHVECTOR_P(0); + LSHVECTOR *b = PG_GETARG_LSHVECTOR_P(1); + TupleDesc tupdesc; + TupleDesc bless; + HeapTuple restuple; + Datum dvalues[2]; + bool nulls[2] = {false, false}; + double sim,sig; + + sim = lsh_compare_internal(a,b,&sig); + PG_FREE_IF_COPY(a,0); + PG_FREE_IF_COPY(b,1); + + if (get_call_result_type(fcinfo,NULL,&tupdesc) != TYPEFUNC_COMPOSITE) + elog(ERROR,"Could not get composite row type to return"); + + bless = BlessTupleDesc(tupdesc); + + dvalues[0] = Float8GetDatum(sim); + dvalues[1] = Float8GetDatum(sig); + restuple = heap_form_tuple(bless,dvalues,nulls); + return HeapTupleGetDatum(restuple); +} + +/* + * This is the actual operator function being accelerated by the gin index. In truth, the index itself + * defines the operator, so the commented out code below emulates the indexes key generation process and + * looks for overlap in the keys between two vectors. In practice, any query that invokes this operator + * will hopefully be going through the index and so doesn't need to evaluate this function. For + * cases where postgresql does a recheck after going through the index, there is no query that doesn't send + * the results of the operator test to a similarity filter. So there is no reason to actually perform + * the overlap test. So we just implement a NOP return that always returns true. + */ +Datum lshvector_overlap(PG_FUNCTION_ARGS) +{ +/* bool res; */ +/* int32 i; */ +/* LSHVECTOR *a = PG_GETARG_LSHVECTOR_P(0); */ +/* LSHVECTOR *b = PG_GETARG_LSHVECTOR_P(1); */ +/* uint32 *bina = (uint32 *)palloc( sizeof(uint32) * lsh_L ); */ +/* uint32 *binb = (uint32 *)palloc( sizeof(uint32) * lsh_L ); */ + +/* lsh_generate_binids(bina,a->items,a->numitems); */ +/* lsh_generate_binids(binb,b->items,b->numitems); */ +/* PG_FREE_IF_COPY(a,0); */ +/* PG_FREE_IF_COPY(b,1); */ + +/* res = false; /\* Assume no overlap *\/ */ +/* for(i=0;iitems,a->numitems); + PG_FREE_IF_COPY(a,0); + *nkeys = lsh_L; + PG_RETURN_POINTER(entries); +} + +Datum lshvector_gin_extract_query(PG_FUNCTION_ARGS) + +{ + LSHVECTOR *a = PG_GETARG_LSHVECTOR_P(0); + int32 *nkeys = (int32 *) PG_GETARG_POINTER(1); + /* StrategyNumber strategy = PG_GETARG_UINT16(2); */ + /* bool **pmatch = (bool **) PG_GETARG_POINTER(3); */ + /* Pointer **extra_data = (Pointer **) PG_GETARG_POINTER(4); */ + /* bool **nullFlags = (bool **) PG_GETARG_POINTER(5); */ + /* int32 *searchMode = (int32 *) PG_GETARG_POINTER(6); */ + Datum *entries = (Datum *)palloc( sizeof(Datum) * lsh_L ); + + lsh_generate_binids_datum(entries,a->items,a->numitems); + PG_FREE_IF_COPY(a,0); + *nkeys = lsh_L; + PG_RETURN_POINTER(entries); +} + +Datum lshvector_gin_consistent(PG_FUNCTION_ARGS) + +{ + bool *check = (bool *) PG_GETARG_POINTER(0); + /* StrategyNumber strategy = PG_GETARG_UINT16(1); */ + /* LSHVECTOR *a = PG_GETARG_LSHVECTOR_P(2); */ + int32 nkeys = PG_GETARG_INT32(3); + /* Pointer *extra_data = (Pointer *) PG_GETARG_POINTER(4); */ + bool *recheck = (bool *) PG_GETARG_POINTER(5); + bool res = false; + int32 i; + + *recheck = false; /* The operator does NOT need to be recalculated, this routine should exactly match */ + for(i=0;i + +#define LSH_IDFSIZE 512 +#define LSH_TFSIZE 64 +#define LSH_MAX_HASHENTRIES 1048576 +#define LSH_MAX_K 31 +#define LSH_MAX_L 1024 +#define LSH_DEFAULT_K 17 +#define LSH_DEFAULT_L 146 + +int32 lsh_k; /* Number of bits in a binid */ +int32 lsh_L; /* Number of binnings */ + +static double lsh_idfweight[LSH_IDFSIZE]; /* Sorted weights least -> most probable for Inverse Document Freq */ +static double lsh_tfweight[LSH_TFSIZE]; /* Sorted weights least -> most probable for Term Frequency */ +static double lsh_weightnorm; /* Normalization of idf weights over raw log(probability) */ +static double lsh_probflip0; /* Significance penalty for hash flips */ +static double lsh_probflip1; +static double lsh_probdiff0; /* Significance penalty for length differences */ +static double lsh_probdiff1; +static double lsh_scale; /* Final scaling for significance scoring */ +static double lsh_addend; +static double lsh_probflip0_norm; +static double lsh_probflip1_norm; +static double lsh_probdiff0_norm; +static double lsh_probdiff1_norm; + +typedef struct { + uint32 hash; + uint32 count; +} IDFEntry; + +static MemoryContext lsh_mem_ctx; +static uint32 lsh_IDFTableMask; /* mask for hash table computation */ +static IDFEntry *lsh_IDFTable = NULL; /* The IDFLookup table */ +bool weights_loaded = false; + +static void update_norms(void) + +{ + int32 i; + double scale_sqrt = sqrt(lsh_scale); + lsh_probflip0_norm = lsh_probflip0 * lsh_scale; + lsh_probflip1_norm = lsh_probflip1 * lsh_scale; + lsh_probdiff0_norm = lsh_probdiff0 * lsh_scale; + lsh_probdiff1_norm = lsh_probdiff1 * lsh_scale; + lsh_weightnorm = lsh_weightnorm / lsh_scale; + for(i=0;itupdesc; + spi_tuptable = SPI_tuptable; + + for(i=0;ivals[i]; + resstring = SPI_getvalue(tuple, spi_tupdesc, 1); /* Column numbers start at 1 */ + resindex = strtol(resstring,NULL,10); + pfree(resstring); + resstring = SPI_getvalue(tuple, spi_tupdesc, 2); + resweight = atof( resstring ); + pfree(resstring); + if (resindex < LSH_IDFSIZE) + lsh_idfweight[resindex] = resweight; + else if (resindex < LSH_IDFSIZE + LSH_TFSIZE) + lsh_tfweight[resindex - LSH_IDFSIZE] = resweight; + else if (resindex == (LSH_IDFSIZE + LSH_TFSIZE)) + lsh_weightnorm = resweight; + else if (resindex == (LSH_IDFSIZE + LSH_TFSIZE + 1)) + lsh_probflip0 = resweight; + else if (resindex == (LSH_IDFSIZE + LSH_TFSIZE + 2)) + lsh_probflip1 = resweight; + else if (resindex == (LSH_IDFSIZE + LSH_TFSIZE + 3)) + lsh_probdiff0 = resweight; + else if (resindex == (LSH_IDFSIZE + LSH_TFSIZE + 4)) + lsh_probdiff1 = resweight; + else if (resindex == (LSH_IDFSIZE + LSH_TFSIZE + 5)) + lsh_scale = resweight; + else if (resindex == (LSH_IDFSIZE + LSH_TFSIZE + 6)) + lsh_addend = resweight; + else { + SPI_finish(); + return false; + } + } + SPI_finish(); + update_norms(); + return true; +} + +void lsh_load_weights(void) + +{ + int32 i; + if (load_weights_from_table()) /* Try to get weights from table */ + return; + + /* Provide some sort of reasonable default */ + for(i=0;icount == 0xffffffff) /* Found an empty slot */ + break; + val = (val + 1) & lsh_IDFTableMask; + } + ptr->hash = hash; + ptr->count = count; +} + +static uint32 get_idflookup_count(uint32 hash) + +{ + uint32 val; + IDFEntry *ptr; + if (lsh_IDFTableMask == 0) + return 0; + val = hash & lsh_IDFTableMask; + for(;;) { + ptr = lsh_IDFTable + val; + if (ptr->count == 0xffffffff) break; /* Is slot empty */ + if (ptr->hash == hash) + return ptr->count; + val = (val + 1) & lsh_IDFTableMask; + } + return 0; /* Entry is not in the table (assume 0 count) */ +} + +/* + * Based on hash and existing idf and tf counts, calculate the final coefficient + * Also calculate the vector length and hashcount + */ +void lsh_calc_weights(LSHVECTOR *vec) + +{ + uint32 i; + LSH_ITEM *ptr; + uint32 idf; + double length = 0.0; + double coeff; + uint32 tf; + uint32 hashcount = 0; + + ptr = vec->items; + for(i=0;inumitems;++i) { + idf = get_idflookup_count(ptr[i].hash); + ptr[i].idf = idf; + tf = ptr[i].tf; + coeff = lsh_idfweight[idf] * lsh_tfweight[ tf - 1 ]; + ptr[i].coeff = coeff; + length += coeff * coeff; + hashcount += tf; + } + vec->length = sqrt(length); + vec->hashcount = hashcount; +} + +/* Load the most common IDF hashes for lookup and weight generation from the table 'idflookup' + * If the table isn't present, return false + * This assumes the existence of a table with (approximately) 1000 rows constructed with + * CREATE TABLE idflookup( hash bigint, lookup integer); + */ +static bool load_idflookup_from_table(void) + +{ + SPITupleTable *spi_tuptable; + TupleDesc spi_tupdesc; + uint64 i,proc; + int32 ret; + char *resstring; + uint32 rescount; + uint32 reshash; + + ret = SPI_connect(); + + if (ret < 0) + elog(ERROR,"lshvector load_idflookup_from_table: SPI_connect returned %d",ret); + + /* Check for the existence of idflookup */ + ret = SPI_execute("SELECT relname from pg_class where relname='idflookup';",true,0); + proc = SPI_processed; + if ((ret != SPI_OK_SELECT)||(proc != 1)) { + elog(WARNING,"lshvector load_idflookup_from_table: No IDF hashes present"); + SPI_finish(); + return false; + } + + ret = SPI_execute("SELECT ALL * from idflookup;",true,0); /* Read(only) all rows from table */ + proc = SPI_processed; + if ((ret != SPI_OK_SELECT)||(proc <= 1)||(proc > LSH_MAX_HASHENTRIES)) { + elog(WARNING,"lshvector load_idflookup_from_table: idflookup has invalid size: IDF hashes not loaded"); + SPI_finish(); + return false; + } + initialize_idflookup_hashtable((uint32)proc); /* Allocate the hashtable to hold entries for each row */ + + spi_tupdesc = SPI_tuptable->tupdesc; + spi_tuptable = SPI_tuptable; + + for(i=0;ivals[i]; + resstring = SPI_getvalue(tuple, spi_tupdesc, 1); /* Column numbers start at 1 */ + reshash = strtoul(resstring,NULL,10); + pfree(resstring); + resstring = SPI_getvalue(tuple, spi_tupdesc, 2); + rescount = strtoul(resstring,NULL,10); + pfree(resstring); + insert_idflookup_hash(reshash,rescount); + } + SPI_finish(); + return true; +} + +void lsh_load_binconfig(void) + +{ /* Load the k and L parameters from the database */ + SPITupleTable *spi_tuptable; + TupleDesc spi_tupdesc; + uint64 proc; + int32 ret; + char *resstring; + HeapTuple tuple; + + ret = SPI_connect(); + + if (ret < 0) + elog(ERROR,"lshvector lsh_load_binconfig: SPI_connect returned %d",ret); + + /* Check for the existence of keyvaluetable */ + ret = SPI_execute("SELECT relname from pg_class where relname='keyvaluetable';",true,0); + proc = SPI_processed; + if ((ret != SPI_OK_SELECT)||(proc != 1)) { + SPI_finish(); + lsh_k = LSH_DEFAULT_K; /* Reasonable defaults if configuration parameters don't exist */ + lsh_L = LSH_DEFAULT_L; + return; + } + + /* Get the 'k' value */ + ret = SPI_execute("SELECT value FROM keyvaluetable WHERE key='k';",true,0); + proc = SPI_processed; + if ((ret != SPI_OK_SELECT)||(proc != 1)) + elog(ERROR,"lshvector lsh_load_binconfig: Could not load 'k' value from keyvaluetable"); + + spi_tupdesc = SPI_tuptable->tupdesc; + spi_tuptable = SPI_tuptable; + + tuple = spi_tuptable->vals[0]; + resstring = SPI_getvalue(tuple,spi_tupdesc, 1); /* First column */ + lsh_k = strtoul(resstring,NULL,10); + pfree(resstring); + + /* Get the 'L' value */ + ret = SPI_execute("SELECT value FROM keyvaluetable WHERE key='L';",true,0); + proc = SPI_processed; + if ((ret != SPI_OK_SELECT)||(proc != 1)) + elog(ERROR,"lshvector lsh_load_binconfig: Could not load 'L' value from keyvaluetable"); + + spi_tupdesc = SPI_tuptable->tupdesc; + spi_tuptable = SPI_tuptable; + + tuple = spi_tuptable->vals[0]; + resstring = SPI_getvalue(tuple,spi_tupdesc, 1); /* First column */ + lsh_L = strtoul(resstring,NULL,10); + pfree(resstring); + SPI_finish(); + + if (lsh_k < 1 || lsh_k > LSH_MAX_K || lsh_L < 1 || lsh_L > LSH_MAX_L) + elog(ERROR,"lshvector lsh_load_binconfig: Invalid k and L settings"); +} + +void lsh_load_lookuptable(void) + +{ + if (lsh_IDFTable != NULL) { + pfree(lsh_IDFTable); + lsh_IDFTable = NULL; + } + + if (load_idflookup_from_table()) + return; + + if (lsh_IDFTable != NULL) { + pfree(lsh_IDFTable); + lsh_IDFTable = NULL; + } + lsh_IDFTableMask = 0; /* Default lookup, always return 0 */ +} + +/* Initialize the weight system, the first time the extension is loaded */ +void lsh_initialize(void) + +{ + lsh_mem_ctx = AllocSetContextCreate(TopMemoryContext, + "IDF weights lookup table", + ALLOCSET_DEFAULT_MINSIZE, + ALLOCSET_DEFAULT_INITSIZE, + ALLOCSET_DEFAULT_MAXSIZE); + + lsh_IDFTable = NULL; + weights_loaded = false; + + lsh_setup_signtable(); +} + +double lsh_compare_internal(LSHVECTOR *a,LSHVECTOR *b,double *sig) + +{ + double res = 0.0; + double dotproduct; + int32 intersectcount = 0; + uint32 hash1,hash2; + LSH_ITEM *aptr,*aend,*bptr,*bend; + int32 t1,t2; + double w1,w2; + uint32 numflip,diff,min,max; + + aptr = a->items; + aend = aptr + a->numitems; + bptr = b->items; + bend = bptr + b->numitems; + + if ((aptr != aend)&&(bptr != bend)) { + hash1 = aptr->hash; + hash2 = bptr->hash; + for(;;) { + if (hash1 == hash2) { + t1 = aptr->tf; + t2 = bptr->tf; + if (t1 < t2) { /* a has the smallest number of terms with same hash */ + w1 = aptr->coeff; /* Use a weight */ + res += w1 * w1; + intersectcount += t1; /* All of a terms are in the intersection, count them */ + } + else { + w2 = bptr->coeff; /* Use b weight */ + res += w2 * w2; + intersectcount += t2; /* All of b terms are in the intersection, count them */ + } + aptr++; + bptr++; + if (aptr == aend) break; + if (bptr == bend) break; + hash1 = aptr->hash; + hash2 = bptr->hash; + } + else if (hash1 < hash2) { + aptr++; + if (aptr == aend) break; + hash1 = aptr->hash; + } + else { /* hash1 > hash2 */ + bptr++; + if (bptr == bend) break; + hash2 = bptr->hash; + } + } + dotproduct = res; + res /= (a->length * b->length); + } + else + dotproduct = res; + + if (a->hashcount < b->hashcount) { + min = a->hashcount; /* Smallest vector is a */ + max = b->hashcount; + } + else { + min = b->hashcount; + max = a->hashcount; + } + diff = max - min; /* Subtract to get a positive difference */ + numflip = min - intersectcount; + *sig = dotproduct - numflip * (lsh_probflip0_norm + lsh_probflip1_norm/max) + - diff * (lsh_probdiff0_norm + lsh_probdiff1_norm/max) + lsh_addend; + return res; +} + diff --git a/Ghidra/Features/BSim/src/lshvector/lshvector--1.0.sql b/Ghidra/Features/BSim/src/lshvector/lshvector--1.0.sql new file mode 100755 index 0000000000..ce1ee6a3d9 --- /dev/null +++ b/Ghidra/Features/BSim/src/lshvector/lshvector--1.0.sql @@ -0,0 +1,107 @@ + + +-- complain if script is sourced in psql, rather than via CREATE EXTENSION +\echo Use "CREATE EXTENSION lshvector" to load this file. \quit + +-- Create user-defined type for feature vector + +CREATE FUNCTION lshvector_in(cstring) +RETURNS lshvector +AS 'MODULE_PATHNAME' +LANGUAGE C STABLE STRICT; +-- Stable because of configurable weights + +CREATE FUNCTION lshvector_out(lshvector) +RETURNS cstring +AS 'MODULE_PATHNAME' +LANGUAGE C IMMUTABLE STRICT; + +CREATE FUNCTION lshvector_recv(internal) +RETURNS lshvector +AS 'MODULE_PATHNAME' +LANGUAGE C STABLE STRICT; +-- Stable because of configurable weights + +CREATE FUNCTION lshvector_send(lshvector) +RETURNS bytea +AS 'MODULE_PATHNAME' +LANGUAGE C IMMUTABLE STRICT; + +CREATE FUNCTION lshvector_hash(lshvector) +RETURNS int8 +AS 'MODULE_PATHNAME' +LANGUAGE C IMMUTABLE STRICT; + +CREATE FUNCTION lsh_load() +RETURNS int4 +AS 'MODULE_PATHNAME' +LANGUAGE C STRICT; + +CREATE FUNCTION lsh_reload() +RETURNS int4 +AS 'MODULE_PATHNAME' +LANGUAGE C STRICT; + +CREATE FUNCTION lsh_getweight(lshvector) +RETURNS float8 +AS 'MODULE_PATHNAME' +LANGUAGE C IMMUTABLE STRICT; + +CREATE TYPE lshvector ( + INTERNALLENGTH = variable, + INPUT = lshvector_in, + OUTPUT = lshvector_out, + RECEIVE = lshvector_recv, + SEND = lshvector_send, + ALIGNMENT = double, + STORAGE = external +); + +CREATE TYPE lshvector_comptype AS ( + sim DOUBLE PRECISION, + sig DOUBLE PRECISION +); + +CREATE FUNCTION lshvector_compare(lshvector,lshvector) +RETURNS lshvector_comptype +AS 'MODULE_PATHNAME' +LANGUAGE C IMMUTABLE STRICT; + +CREATE FUNCTION lshvector_overlap(lshvector,lshvector) +RETURNS bool +AS 'MODULE_PATHNAME' +LANGUAGE C STABLE STRICT; + +CREATE FUNCTION lshvector_gin_extract_value(lshvector,internal) +RETURNS internal +AS 'MODULE_PATHNAME' +LANGUAGE C STABLE STRICT; + +CREATE FUNCTION lshvector_gin_extract_query(lshvector,internal,int2,internal,internal,internal,internal) +RETURNS internal +AS 'MODULE_PATHNAME' +LANGUAGE C STABLE STRICT; + +CREATE FUNCTION lshvector_gin_consistent(internal, int2, lshvector, int4, internal, internal, internal, internal) +RETURNS bool +AS 'MODULE_PATHNAME' +LANGUAGE C IMMUTABLE STRICT; + +CREATE OPERATOR % ( + LEFTARG = lshvector, + RIGHTARG = lshvector, + PROCEDURE = lshvector_overlap, + COMMUTATOR = '%', + RESTRICT = contsel, + JOIN = contjoinsel +); + +CREATE OPERATOR CLASS gin_lshvector_ops +FOR TYPE lshvector USING gin +AS + OPERATOR 1 % (lshvector,lshvector), + FUNCTION 1 btint4cmp (int4,int4), + FUNCTION 2 lshvector_gin_extract_value (lshvector,internal), + FUNCTION 3 lshvector_gin_extract_query (lshvector,internal,int2,internal,internal,internal,internal), + FUNCTION 4 lshvector_gin_consistent (internal,int2,lshvector,int4,internal,internal,internal,internal), + STORAGE int4; diff --git a/Ghidra/Features/BSim/src/lshvector/lshvector.control b/Ghidra/Features/BSim/src/lshvector/lshvector.control new file mode 100755 index 0000000000..7f58d86164 --- /dev/null +++ b/Ghidra/Features/BSim/src/lshvector/lshvector.control @@ -0,0 +1,6 @@ +# Locality Sensitive Hashing extension +comment = 'a feature vector type and a locality sensitive hashing index' +default_version = '1.0' +module_pathname = '$libdir/lshvector' +superuser = false +relocatable = true diff --git a/Ghidra/Features/BSim/src/main/help/help/TOC_Source.xml b/Ghidra/Features/BSim/src/main/help/help/TOC_Source.xml new file mode 100755 index 0000000000..ba54898ec6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/TOC_Source.xml @@ -0,0 +1,175 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Ghidra/Features/BSim/src/main/help/help/shared/languages.css b/Ghidra/Features/BSim/src/main/help/help/shared/languages.css new file mode 100644 index 0000000000..657f4bfe30 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/shared/languages.css @@ -0,0 +1,25 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +/* + This file contains non-Ghidra style sheet markup. This file will be loaded in addition to + DefaultStyle.css. +*/ + +div.informalexample { margin-left: 50px; margin-top: 10px; } +dd { margin-bottom: 20px; } +dd p { margin-top: 5px; margin-left: 10px; } +span.term { font-family:times new roman; font-size:14pt; font-weight:bold; } +span.redtext { color:#CC0033; } diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/BSimOverview.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/BSimOverview.html new file mode 100644 index 0000000000..e9bf750109 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/BSimOverview.html @@ -0,0 +1,197 @@ + + + + + + + + BSim Database + + + + + + + + + + +
+
+
+
+

BSim Database

+
+
+
+ +
+

Quick Reference Links

+ +
+ +
+
+ +
+
+
+
+

Overview

+
+
+
+ +

Welcome to Ghidra's BSim (Behavioral Similarity) Database. This database technology is + designed to allow reverse engineers to ingest metadata about previously analyzed binary + executables to a central server or local database, which can then be queried in the + course of analyzing new, + unknown, executables to quickly discover previously seen functions and libraries.

+ +

The primary record ingested into the database describes a single function. The most + novel aspects of the database are that:

+ +
+
+
    +
  • Queries are tolerant of variations in the compilation of the + function.
    • + +
  • All records are indexed for quick queries. (even for very large + collections)
    • +
+
+
+ +

The primary feature set used for indexing a function is extracted from a concise + description of the data-flow of the function, not the explicit encoding of the machine + instructions. The data-flow description is a graph-based (abstract syntax tree) + representation, based on Ghidra's intermediate representation language, p-code, and is + generated by the Ghidra decompiler. The resulting function descriptions are normalized to + minimize the impact of variations due to:

+ +
+
+
    +
  • Equivalent machine instructions
    • + +
  • Storage location (registers, stack, memory)
    • + +
  • Instruction order
    • + +
  • Many forms of compiler transformation
    • + +
  • Even some forms of deliberate obfuscation.
    • +
+
+
+ +

Records are indexed using current Text Retrieval strategies, which allow "nearest + neighbor" queries. The feature set of an unknown function being queried does not have to + exactly match the features of a "hit" in the database, but only a configurable percentage + of them. This supplies an additional level of tolerance of "functional difference" on top + of the tolerance of "functionally equivalent" variations provided by the decompiler. In + other words, there can be some amount of true change in the underlying source code, and the + query may still be able to find a match.

+ +

Queries are quick: For a single function, results typically come back in microseconds, + even for a database containing millions of functions.

+
+ +
+
+
+
+

Overview of + Tools

+
+
+
+ +

A BSim Database is built on top of one of three technologies: PostgreSQL, + local H2 database, or Elasticsearch. + PostgreSQL is a robust, production capable, server that supports multiple simultaneous + connections and is extremely fault tolerant. Elasticsearch is a scalable search engine that + allows a database to be distributed across an entire cluster of machines. + The local H2 database support is provided for convenience and use with small personal + collections. For any of these options, this distribution includes specific reverse + engineering extensions and clients that provide the following capabilities.

+ +
+
+
    +
  • + Integration with a Ghidra Server or local project: + +
    +
      +
    • Ingest can be with respect to a Ghidra repository + from either a Ghidra Server or local project.
      • + +
    • Query results can refer to executables within a + repository.
      • + +
    • Easy command-line ingests using the bsim command script
      • +
    +
    +
    • + +
  • + Client as a Ghidra Plug-in: + +
    +
      +
    • Ghidra includes a plug-in client that integrates a query + dialog and results windows directly into the main code browser.
      • +
    +
    +
    • + +
  • + Query API: + +
    +
      +
    • Ghidra includes a Java API to the BSim server so that + queries (and potentially ingest) can be incorporated into analyst scripts. The + API marshals queries and results between an active Ghidra session and a BSim + server.
      • +
    +
    +
    • +
+
+
+ +
+

Note

+ +

The PostgreSQL server software is currently only supported for the Linux and MacOS + architectures. Elasticsearch server software must be obtained separately. Small local + file-based databases are supported on all platforms via an embedded H2 database + engine. The BSim client + software is supported on all platforms and can connect to servers on a different + architecture.

+
+
+
+ + diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/CommandLineReference.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/CommandLineReference.html new file mode 100644 index 0000000000..c3f3bfa17a --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/CommandLineReference.html @@ -0,0 +1,820 @@ + + + + + + + + Command-Line Utility Reference + + + + + + + + + +
+
+
+
+

Command-Line Utility + Reference

+
+
+
+ +
+
+
+
+

bsim_ctl

+
+
+
+ +
+
+
+    bsim_ctl start           </datadir-path [auth=pki|password|trust] [--noLocalAuth] [cafile=</cacert-path>] [dn=".."]
+    bsim_ctl stop            </datadir-path> [--force]
+    bsim_ctl adduser         </datadir-path> <username> [dn=".."]
+    bsim_ctl dropuser        </datadir-path> <username>
+    bsim_ctl resetpassword   <username>
+    bsim_ctl changeauth      </datadir-path> [auth=pki|password|trust] [--noLocalAuth] [cafile=</cacert-path>] [dn=".."]
+    bsim_ctl changeprivilege <username> admin|user
+    
+    Global Options:
+        port=<portnum>
+        user=<username>
+        cert=</certfile-path>
+
+
+
+ +

bsim_ctl is a command-line utility for + starting and stopping a BSim server using the PostgreSQL back-end that is prepackaged with + the Ghidra distribution. All commands must be run on the machine hosting the server. + Optional parameters for a given command are indicated by square brackets '[' and ']'. + Options with an '=' character require a user specified value. If the value string requires + space characters, it should be enclosed in double quotes.

+ +
+
+
+
start
+ +
+

Initializes and starts a PostgreSQL server. The command-line must include a path + to the data directory for the server, which must exist. If a server had run + previously and populated this directory, this command simply restarts the server + using the preexisting data and configuration; otherwise, a new database is + initialized. The user performing the initial start is automatically added to the + database with admin privileges.

+ +

During a restart, any authentication options (with the exception of the global + cert= option) are unnecessary and will + be ignored. The PostgreSQL server will be restarted with the already established + settings. To actually change the settings, use the changeauth command before restarting.

+ +

auth=type - specifies the authentication type (pki | + password | trust) for a new database: trust for no authentication, password for password authentication, and pki for authentication using public key certificates. + With the pki setting, both the cafile= and the dn= options also need to be provided; additionally + the cert= option must be provided unless + the --noLocalAuth option is also + given.

+ +

--noLocalAuth - used together with + the auth= option causes + authentication to not be required for local connections, i.e. localhost.

+ +

cafile=/cafile-path - specifies an absolute path to a + certificate authority file and is required for auth=pki. This file should contain the + certificates the PostgreSQL server will use to authenticate in PEM format + concatenated together.

+ +

dn=name - specifies the Distinguished Name for the admin + user and is required for auth=pki.

+ +

port=portnum - specifies the port the PostgreSQL server will + listen on. For port numbers other than the default 5432, URLs and other + command-lines must explicitly specify the port, when connecting to the server. This + option only effects the initial start of a server. For subsequent (re)starts this + option is ignored, and the server will continue to listen on the same port + specified in the initial start. Use changeauth to change the port of a server after + its initial start.

+
+ +
stop
+ +
+

Stops a currently running PostgreSQL server. The path to the actively used data + directory must be provided. By default, shutdown will wait until existing + connections to the database have been closed.

+ +

--force - causes existing + connections to be forcibly closed and the PostgreSQL server to shut down + immediately.

+
+ +
adduser
+ +
+

Give a new user permission to access the PostgreSQL server. The path to the + actively used data directory and a single username must be specified. The server + must be running. New users are given user + (read-only) privileges, unless a subsequent changeprivilege command is used.

+ +

dn=name - specifies the Distinguished Name of the new user, + which is required if the database enabled auth=pki. This option can be used to provide a + Distinguished Name to a preexisting user, if the PostgreSQL server's authentication + strategy is changed.

+
+ +
dropuser
+ +
+

Remove access to the PostgreSQL server for a specific user. The path to the + actively used data directory and a single username must be specified. The server + must be running.

+
+ +
changeauth
+ +
+

Change the configuration of a previously initialized PostgreSQL server. The path + to the server's data directory must be specified. The server must not currently be + running to use this command, which only takes effect after a restart. Options have + the same meaning as for the start + command.

+ +

port=portnum - changes the port the PostgreSQL server will + listen on. If this option is not present, the server will continue to listen on the + same port.

+ +

auth=type - changes the authentication type (pki | + password | trust) used by the PostgreSQL server. No change is made if the + option is not present. If the option is present, omitting the --noLocalAuth causes local connections to require + authentication. This command does not affect the presence or absence of passwords + or Distinguished Names for existing users.

+ +

dn=name - specifies the Distinguished Name for the admin + user and is required for auth=pki.

+
+ +
resetpassword
+ +
+

Reset the password for a user. A single user must be specified, and the + PostgreSQL server must be running. The password will be reset to 'changeme'.

+
+ +
changeprivilege
+ +
+

Change access privilege for a user. A single user must be specified followed by + admin or user, and the PostgreSQL server must be + running.

+
+ +
--Global + Options--
+ +
+

These options apply to all the bsim_ctl commands that connect to an active + PostgreSQL server: start, adduser, dropuser, resetpassword, and changeprivilege.

+ +

port=portnum - specifies the port on which to connect with + the PostgreSQL server.

+ +

user=username - specifies a user name to use when connecting + to the PostgreSQL server.

+ +

cert=/certfile-path - provides the absolute file path to the + user's certificate when connecting to a PostgreSQL server that requires PKI + authentication.

+
+
+
+
+
+ +
+
+
+
+

bsim

+
+
+
+ +
+
+
+    bsim createdatabase  <bsimURL> <config_template> [name="<name>"] [owner="<owner>"] [description="<text>"] [--nocallgraph]
+    bsim setmetadata     <bsimURL> [name="<name>"] [owner="<owner>"] [description="<text>"]\n" + 
+    bsim addexecategory  <bsimURL> <category_name> [--date]
+    bsim addfunctiontag  <bsimURL> <tag_name>
+    bsim dropindex       <bsimURL>
+    bsim rebuildindex    <bsimURL>
+    bsim prewarm         <bsimURL>
+    bsim generatesigs    <ghidraURL> </xmldirectory> config=<config_template> [--overwrite]
+    bsim generatesigs    <ghidraURL> </xmldirectory> bsim=<bsimURL> [--commit] [--overwrite]
+    bsim generatesigs    <ghidraURL> bsim=<bsimURL>
+    bsim commitsigs      <bsimURL> </xmldirectory> [md5=<hash>] [override=<ghidraURL>]
+    bsim generateupdates <ghidraURL> </xmldirectory> config=<config_template> [--overwrite]
+    bsim generateupdates <ghidraURL> </xmldirectory> bsim=<bsimURL> [--commit] [--overwrite]
+    bsim generateupdates <ghidraURL> bsim=<bsimURL>
+    bsim commitupdates   <bsimURL> </xmldirectory>
+    bsim listexes        <bsimURL> [md5=<hash>] [name=<exe_name>] [arch=<languageID>] [compiler=<cspecID>] [sortcol=<column_name>] [limit=<exe_count>] [--includelibs]
+    bsim getexecount     <bsimURL> [md5=<hash>] [name=<exe_name>] [arch=<languageID>] [compiler=<cspecID>] [--includelibs]
+    bsim delete          <bsimURL> [md5=<hash>] [name=<exe_name> [arch=<languageID>] [compiler=<cspecID>]]
+    bsim listfuncs       <bsimURL> [md5=<hash>] [name=<exe_name> [arch=<languageID>] [compiler=<cspecID>]] [--printselfsig] [--callgraph] [--printjustexe] [maxfunc=<max_count>]
+    bsim dumpsigs        <bsimURL> </xmldirectory> [md5=<hash>] [name=<exe_name> [arch=<languageID>] [compiler=<cspecID>]]
+    
+    Global options:
+        user=<username>
+        cert=<certfile-path>
+
+
+
+ +

See “Ghidra and BSim + URLs” below for details about specifying ghidraURL and bsimURL + properly. See “Database + Configuration” for guidance on the various BSim Databases which are + supported.

+ +

bsim is a command-line utility for + managing the generation and ingest of BSim signatures and metadata. Depending on the + subcommand, it connects to a Ghidra Server and/or a BSim database server. A ghidraURL refers to Ghidra Server or local project using the + ghidra: protocol, while bsimURL refers to a BSim database server with the appropriate + postgresql:, https:, or file: protocol specified. The elastic: protocol is equivalent to and may be used in + place of the https: protocol.

+ +
+
+
+
createdatabase
+ +
+

Creates a new empty repository. A URL and configuration template (config_template) is required. The new database name + is taken from the path element of the URL.

+ +

Supported configuration templates (config_template) are defined within the Ghidra + installation in XML form. The following configurations are currently defined: + (large_32, medium_32, medium_64, medium_cpool, + medium_nosize).

+ +

name= - specifies a formal, more + descriptive, name for the repository that can be used for the BSim client + display.

+ +

owner= - gives a descriptive name + for the owner of the repository and/or the data it will contain.

+ +

description= - specifies a short + string describing the intended contents of the new repository.

+ +

--nocallgraph=yes/no - disables storing call relationships between + ingested functions. Default is to store call relationships.

+
+ +
setmetadata
+ +
+

Change the global name, owner, or description metadata associated with a BSim server. A + BSim server URL is required.

+ +

name= - specifies a formal, more + descriptive, name for the repository that can be used for the BSim client + display.

+ +

owner= - gives a descriptive name + for the owner of the repository and/or the data it will contain.

+ +

description= - specifies a short + string describing the intended contents of the new repository.

+
+ +
addexecategory
+ +
+

Specify a new executable category to be included with generated metadata. A BSim + server URL and the name of the new category are required. This only affects future + ingest commands. Executables that have already been ingested are unaffected, + although they can be adjusted with an updaterepo command.

+ +

date - indicates the new category + holds date/time information.

+
+ +
addfunctiontag
+ +
+

Specify a new function tag to be included with generated metadata. A BSim server + URL and the name of the new tag are required. This only affects future ingest + commands. Functions that have already been ingested are unaffected, although they + can be adjusted with an updaterepo + command.

+
+ +
dropindex
+ +
+

Delete the main signature index from a BSim repository (in preparation for new + ingest). A BSim repository URL is required. Normal queries will not complete or + will be extremely slow.

+ +

NOTE: Not supported by H2 file database

+
+ +
rebuildindex
+ +
+

Recreate the main signature index (that had previously been dropped) for a BSim + repository. A BSim server URL is required. After this command completes, normal + function queries should be fast.

+ +

NOTE: Not supported by H2 file database

+
+ +
prewarm
+ +
+

Instruct a restarted BSim server to preload pages from the main signature index + and function table into RAM. This avoids slow random access disk reads on initial + queries. A BSim server URL is required.

+ +

NOTE: Not supported by Elasticsearch or H2 file databases

+
+ +
generatesigs
+ +
+

Generates function signatures and metadata for all program files retrieved from + a Ghidra Server repository or project as specified by a Ghidra URL. The generated + signatures may be retained as XML "sigs_" files within a specified XML storage + directory and/or commited to a specified BSim database specified with the bsim=bsimURL option. If an XML storage directory is not + specified, a BSim URL must be specified to which the data will be committed.

+ +

The config=config-template option may be specified when generating + XML "sigs_" signature files in the absence of a BSim database (see + createdatabase for supported configurations). The generated files + will be written to the specified XML storage directory. Creation of the signature + files can also be achieved by specifying the bsim=bsimURL + option instead of the config= option.

+ +

The --overwrite option may be specified when an XML storage directory has also been + specified to allow conflicting signature files to be overwritten.

+ +

The --commit option may be specified when a BSim URL has also been specified to allow + generated signatures to be committed to the BSim database. This option is implied + when a BSim URL has been specified without an XML storage directory.

+
+ +
commitsigs
+ +
+

Commit previously generated signatures and metadata (see + signaturerepo) to a BSim repository. A URL specifying the BSim + repository and a path to a directory containing the "sigs_" XML files to commit are + required.

+ +

override=ghidraURL - causes any Ghidra repository/project URL, + describing the storage repository and path of executables that was recorded in the + "sigs_" XML files during signature generation, to be overridden during the commit + operation with the specified Ghidra URL.

+
+ +
generateupdates
+ +
+

Generates updated function metadata for program files from a Ghidra Server + repository or project, as specified by a Ghidra URL, which previously had signature + and metadata generated (see generatesigs). Only metadata: names, + function tags, categories, etc. are changed. Signatures are not affected. The + generated updates may be retained as XML "update_" files within a specified XML + storage directory and/or commited to a specified BSim database specified with the + bsim=bsimURL option. If an XML storage directory is not + specified, a BSim URL must be specified to which the data will be committed.

+ +

The config=config-template option may be specified when generating + XML "update_" files in the absence of a BSim database (see + createdatabase for supported configurations). The generated files + will be written to the specified XML storage directory. Creation of the update + files can also be achieved by specifying the bsim=bsimURL + option instead of the config= option.

+ +

The --overwrite option may be specified when an XML storage directory has also been + specified to allow conflicting update files to be overwritten.

+ +

The --commit option may be specified when a BSim URL has also been specified to allow + generated updates to be committed to the BSim database. This option is implied when + a BSim URL has been specified without an XML storage directory.

+
+ +
commitupdates
+ +
+

Update a BSim repository with previously generated update metadata (see + generateupdates). A URL specifying the BSim repository and a path + to a directory containing the "update_" XML files to commit are required.

+
+ +
listexes
+ +
+

List all executable program records within a specified BSim database repository + which satisfy the specified criteria. A BSim URL specifying the repository must be + provided, and one of two options, md5= or name=, that indicate the specific executable must + also be given. All matching executable records will be listed.

+ +

md5=32-hexdigits - specifies an executable via its MD5 + checksum.

+ +

name= - specifies an executable + name which may match one or more executable records.

+ +

arch= - specifies an architecture + as a Ghidra processor id which will be used to filter executables.

+ +

compiler= - specifies a compiler + specification id which will be used to filter executables.

+ +

sortcol=column - Indicates which display column should be used + to sort the results (MD5 | NAME; default: + MD5).

+ +

limit=max_count - specifies the maximum number of executables + to be listed which match the search criteria (default=20, a value of 0 indicates no + limit).

+ +

--includelibs - If specified, executable + records which correspond to a referenced Library will be included. Such records + have a fabricated MD5 which is based on its name.

+
+ +
getexecount
+ +
+

Get the total number of executable program records within a specified BSim + database repository which satisfy the specified criteria. A BSim URL specifying the + repository must be provided, and one of two options, md5= or name=, that indicate the specific executable must + also be given. All matching executable records will be listed.

+ +

md5=32-hexdigits - specifies an executable via its MD5 + checksum.

+ +

name= - specifies an executable + name which may match one or more executable records.

+ +

arch= - specifies an architecture + as a Ghidra processor id which will be used to filter executables.

+ +

compiler= - specifies a compiler + specification id which will be used to filter executables.

+ +

--includelibs - If specified, executable + records which correspond to a referenced Library will be included. Such records + have a fabricated MD5 which is based on its name.

+
+ +
delete
+ +
+

Remove all records associated with a specific executable from a BSim repository. + A BSim URL specifying the repository must be provided, and one of two options, + md5= or name=, that indicate the specific executable must + also be given. All associated executable and function records are removed. + If an executable cannot be uniquely identified an error will result. +

+ +

md5=32-hexdigits - specifies the executable via its MD5 + checksum.

+ +

name= - specifies an executable + name which may match one or more executable records.

+ +

arch= - specifies an architecture + as a Ghidra processor id, when the name option is not enough to uniquely specify the + executable.

+ +

compiler= - specifies a compiler + id string, when the name option is + not enough to uniquely specify the executable.

+
+ +
listfuncs
+ +
+

List all function records associated with a specific executable from a BSim + repository. A BSim URL specifying the repository must be provided, and one of two + options, md5= or name=, that indicate the specific executable must + also be given. All associated executable and function records are listed. If an + executable cannot be uniquely identified an error will result.

+ +

md5=32-hexdigits - specifies the executable via its MD5 + checksum.

+ +

name= - specifies an executable + name which may match one or more executable records.

+ +

arch= - specifies an architecture + as a Ghidra processor id, when the name option is not enough to uniquely specify the + executable.

+ +

compiler= - specifies a compiler + id string, when the name option is + not enough to uniquely specify the executable.

+ +

--printselfsig - If specified, each + function listed will be prefixed by a calculated self-significance score. This value is + expressed as a decimal value.

+ +

--callgraph - If specified, a list + of all library functions called by the identified executable will be listed after + the function list.

+ +

--printjustexe - If specified, only a + summary of the executable will be displayed. If --callgraph was + also specified only the called libraries will be listed and not the specified + functions.

+ +

maxfunc=max_count - specifies the maximum number of functions to + be listed which correspond to the identified executable (default=1000, a value of 0 + indicates no limit).

+
+ +
dumpsigs
+ +
+

Dump signature and metadata from a BSim repository for a specific executable to + a "sigs_" XML file. A BSim server URL and a path to a directory where the new file + will be stored must be given. One of two options, md5= or name=, that specify the particular executable + must also be given. If an executable cannot be uniquely identified an error will result.

+ +

md5=32-hexdigits - specifies an executable via its MD5 + checksum.

+ +

name= - specifies an executable + name which may match one or more executable records.

+ +

arch= - specifies an architecture + as a Ghidra processor id, when the name option is not enough to uniquely specify the + executable.

+ +

compiler= - specifies a compiler + specification id, when the name option is not enough to uniquely specify the + executable.

+ +
+ +
--Global + Options--
+ +
+

These options apply to all bsim + commands.

+ +

user=name - specifies a user to masquerade as when connecting + to the server.

+ +

cert=path - provides a path to the user's certificate when + connecting to a server that requires PKI authentication.

+
+
+
+
+
+
+ +
+
+
+
+

Ghidra and BSim URLs

+
+
+
+ +

Ghidra utilizes Universal Resource Locators (URLs) to identify both Ghidra + Server/Project Repositories and BSim Databases. See the corresponding sections + below for specific formatting details. It is important to note that local ghidra and + file URLs never include a double-slash after the protocol (i.e, "://").

+ +
+
+
+
+

Ghidra Server/Project + Repository URLs

+
+
+
+ +

BSim command-line tools, as well as the Ghidra GUI, utilize a URL to specify the + location of a remote Ghidra Server repository or a local Ghidra Project. Both cases work in + a very similar fashion other than the format of the URL and potential limitations of a + local Project URL. Use of a Ghidra Server repository and corresponding URLs is preferred + since any Ghidra URL metadata added to a shared BSim database has the ability to be + accessed by other users, while a local Ghidra Project URL is very limited in its visibility + and path validity on other systems. For this reason, use of a local Ghidra Project URL + should be restricted to use with a local H2 BSim Database file.

+ +

The format of a remote Ghidra Server URL is distinctly different from a + Local Ghidra Project URL. These URLs have the following formats:

+ +

Remote Ghidra Server Repository
+

+ +
+ + + + +
ghidra://<hostname>[:<port>]/<repository_name>[/<folder_path>]
+
+ +

If the default Ghidra Server port (1111) is in use it need not be specified with URL. + The hostname may specify either a Fully Qualified Domain Name (FQDN, e.g., + host.abc.com) or IP v4 Address (e.g., 1.2.3.4).

+ Local Ghidra Project
+ + +
+ + + + +
ghidra:[/<directory_path>]/<project_name>[?/<folder_path>]
+
+ +

For local project URLs, the absolute directory path containing the project + *.gpr locator file must be specified with the project name. The project name + should exclude any .gpr/.rep suffix. Only the '/' character should be used as a + directory separator. In addition, when running on Windows, the directory path should + include its drive desigation preceeded by a '/' (e.g., ghidra:/C:/mydir/myproject?/folderA/folderB).

+
+ +
+
+
+
+

BSim Database URLs

+
+
+
+ +

BSim command-line tools utilize a URL to specify the type and specific details required + to establish a connection to a specific BSim Database. Within the Ghidra GUI the database + details are not specified using a URL and is done using an interactive form. Each BSim + database type has a distinct URL format:

+ +
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
Database TypeURL Format
PostgreSQLpostgresql://<hostname>[:<port>]/<dbname>
Elasticsearchhttps://<hostname>[:<port>]/<dbname>
Elasticsearchelastic://<hostname>[:<port>]/<dbname>
H2 Filefile:[/<directory_path>]/<dbname>
+
+ +

The use of the https and elastic is equivalent.

+ +

For local file URLs, the absolute path the H2 database *.mv.db file + must be specified without the *.mv.db extension. Only the '/' character should be + used as a directory separator. In addition, when running on Windows, the directory path + should include its drive desigation preceeded by a '/' (e.g., file:/C:/mydir/mydb).

+
+
+ + diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/DatabaseConfiguration.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/DatabaseConfiguration.html new file mode 100644 index 0000000000..8b592033d4 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/DatabaseConfiguration.html @@ -0,0 +1,993 @@ + + + + + + + + Database Configuration + + + + + + + + + + +
+
+
+
+

Database Configuration

+
+
+
+ +
+
+
+
+

Overview

+
+
+
+ +

The server for the BSim Database is distinct from the traditional Ghidra server, + although for many use cases it is convenient to have both running and view the BSim server + as a loosely coupled extension to the base Ghidra Server. In terms of start-up, shutdown, + and configuration however, the two servers are completely separate.

+ +

There are two choices for deploying a shared server for the BSim Database: PostgreSQL or + Elasticsearch. In addition, a local file-based database may be employed which utilizes an + integrated H2 Database engine. This file-based database is intended for smaller datasets + and its use is limited to a single process.

+ +

PostgreSQL software, including the extension necessary for BSim signature indexing, + comes prepackaged with the Ghidra distribution. It runs on a single host and makes + efficient use of whatever CPU, memory, and disk resources are made available to it. + PostgreSQL is a highly robust and capable server that should perform well on minimally + configured workstations up to high-end production hardware.

+ +

An Elasticsearch BSim plug-in is included with the Ghidra distribution, but the core + server software must be obtained separately by the database administrator. Elasticsearch is + a scalable text search and analytics database. It automatically distributes itself across + machines in a cluster, allowing individual database queries and requests to be serviced in + parallel. Support for BSim in Elasticsearch should still be considered in prototype, but + all major functionality has been implemented, and the BSim schema takes full advantage of + Elasticsearch as a distributed database.

+ +

BSim clients included in the base Ghidra distribution can interface to any of these + databases.

+
+ +
+
+
+
+

Server + Configuration

+
+
+
+ +
+
+
+
+

PostgreSQL Configuration

+
+
+
+ +

The base Ghidra distribution comes with the PostgreSQL software and the extensions + necessary for supporting a BSim database. The PostgreSQL server is most easily managed + using the bsim_ctl command-line script. When + bsim_ctl start is run for the first time (see + below), the PostgreSQL software is unpacked, depending on the host OS, to either

+ +
+ + + + + + + + +
$(ROOT)/Ghidra/Features/BSim/os/linux64/postgresql + OR
$(ROOT)/Ghidra/Features/BSim/os/osx64/postgresql
+
+ +

BSim will not operate with PostgreSQL without the Ghidra specific extensions, but + otherwise the provided installation is standard. It can be configured just like any other + stand-alone PostgreSQL server. PostgreSQL is highly configurable, and there are no direct + restrictions on modifying the configuration values. A default configuration is provided + with this installation that has been tuned specifically for the BSim Database + application, so in practice there may be little reason to modify it. But there are a few + standard configuration values for the server that might need adjusting. These do impact + important aspects of the server, like the amount of memory allocated to the server and + access restrictions.

+ +
+
+
+
+

Starting and Stopping the + Server

+
+
+
+ +

The basic start-up and shut-down is accomplished with the same command-line script, + which takes either the keyword start or + stop as the first parameter. The second + parameter must be an absolute path to the chosen data directory.

+ +
+ + + + + + + + + + + + + + + + +
$(ROOT)/support/bsim_ctl start + /path/to/datadir
$(ROOT)/support/bsim_ctl start /path/to/datadir + port=8000
$(ROOT)/support/bsim_ctl stop + /path/to/datadir
$(ROOT)/support/bsim_ctl stop /path/to/datadir + force
+
+ +

The data directory should already exist and should initially not contain any files. + The first time a server is started for a particular data directory, a large number of + configuration files and other sub-directories associated with the PostgreSQL server + will automatically be created. Upon subsequent restarts the existing configuration will + be reused.

+ +

The start command can take an optional + port= parameter. This can be used to specify + a non-standard port for the PostgreSQL server to listen on. In this case, any + subsequent reference to the BSim server, in the Ghidra client, or with the bsim command described below, must specify the port. + When using the bsim command, a + non-default port must be explicitly specified with the BSim postgresql:// URL (see “Ghidra and BSim URLs” for more + details).

+ +

The stop command can take the keyword + force as an optional parameter. Without + this, the shutdown of the server will wait until all currently connected clients finish + their sessions. Adding this parameter will cause all clients to be disconnected + immediately, rolling back any transactions, and the server will shutdown + immediately.

+
+ +
+
+
+
+

Security and + Authentication

+
+
+
+ +

BSim makes use of PostgreSQL security mechanisms to enforce privileges and + authenticate users. The bsim_ctl command + wraps the subset of functionality described here, but other adjustments are possible by + connecting directly to the server and issuing SQL commands.

+ +

The PostgreSQL server, as configured for BSim, only accepts connections via SSL, so + communications in transit are always encrypted regardless of the authentication + settings.

+ +

PostgreSQL uses the concept of roles to grant + access privileges based on particular users. Generally, a user's role is determined by + the username used to establish the connection. + For BSim, each user role is granted one of two privilege levels: user, which allows read-only access to the server for + normal queries, and admin, which + additionally allows database creation, ingest, update, and deletion.

+ +

BSim supports three different authentication methods, when connecting as a client or + during database ingest and maintenance. This method is established for a server by the + initial start command.

+ +
+
+
+
trust
+ +
+

bsim_ctl start /path/to/datadir + auth=trust

+ +

This is currently the default. No authentication is performed and privilege + is granted based on the user name presented. Masquerading is possible.

+
+ +
password
+ +
+

bsim_ctl start /path/to/datadir + auth=password

+ +

Users are authenticated via password. A default password 'changeme' is + established when the new user is created. Passwords can be changed by the user + from the BSim client or can be reset by an administrator via the resetpassword command.

+
+ +
pki
+ +
+

bsim_ctl start /path/to/datadir auth=pki + ca=/path/to/rootcert

+ +

Users are authenticated by PKI certificates. Upon initialization, the BSim + server must be provided (via the ca= option) a file containing the public keys + for the certificate authorities used to issue user's certificates. The file + consists of the authoritative certificates in PEM format concatenated + together.

+ +

BSim users must register their certificate with the Ghidra client using the + Edit->Set PKI Certificate... menu + option from the Project dialog. The BSim client will automatically submit the + certificate to a server that requests it, and the password to unlock it will be + requested as needed. This is the same mechanism used to a access a PKI + protected Ghidra server, and if a user needs access to both a BSim server and + Ghidra server that are PKI protected, the servers should probably be configured + with the same certificate authorities so that they will accept the same + certificate from the user.

+ +

With PKI authentication enabled, at the time a new user role is established + with the server, the X.509 Distinguished Name, as bound to the user's + certificate, must be associated with the user name via the dn= option. See “Adding Users to the + Database”.

+
+
+
+
+ +

The authentication method should be established once, the first time the start command is issued for the server on an + empty data directory. Subsequent restarts of the server will not change these settings. + If the settings really need to be changed, the changeauth command can be issued. It takes the same + options as the start command and can only + be run if the server is shutdown first.

+ +
+ + + + +
$(ROOT)/support/bsim_ctl changeauth + /datadir/path auth=password
+
+ +

Using the changeauth command on a + server with an established set of users will likely require other disruptive changes to + create passwords or associate Distinguished Names with users, if they didn't exist + before.

+ +

If it is determined that only the database administrators have OS level, local, + access to the server's host machine, they can choose to use the noLocalAuth option as part of the start or changeauth commands. This disables authentication for + users connecting to the server by the 'localhost' interface. This may facilitate the + use of scripts for ingest etc., where working with passwords is cumbersome. + Authentication is still enforced for any remote connection.

+
+ +
+
+
+
+

Adding Users to the Database

+
+
+
+ +

The username used to start the server for the first time, causing the initialization + of the data directory, becomes the administrator for that server. No other + username/role is initially known to the server. New usernames/roles can be added to the + server using the following command:

+ +
+ + + + + + + + +
$(ROOT)/support/bsim_ctl adduser username
$(ROOT)/support/bsim_ctl adduser username dn="C=US,ST=MD,CN=Firstname User"
+
+ +

If password authentication has been set for the server, the new user's password will + initially be set to 'changeme'. If PKI authentication has been set for the server, The + Distinguished Name, as bound to the new user's certificated must be provided when + issuing the adduser command, via the + dn= option. The Distinguished Name must + be presented as a string containing a comma separated sequence of attribute/value pairs + that uniquely identifies a certificate. Currently, the Common Name (CN=) is the only + attribute inspected by the PostgreSQL server, so other attributes can be omitted.

+ +

New users are by default only given user permissions, meaning that they can only place + queries to the database and cannot ingest, update, or delete data. The new user can be + given admin privileges (by an existing + administrator) by issuing the command:

+ +
+ + + + +
$(ROOT)/support/bsim_ctl changeprivilege username admin
+
+
+ +
+
+
+
+

Additional + Configuration

+
+
+
+ +

The relevant configuration files are at the top level of the data directory:

+ +
+ + + + + + + + +
postgresql.conf
pg_hba.conf
+
+ +

The most important configuration parameters in postgresql.conf are:

+ +
+
+
+
shared_buffers
+ +
+

This controls the amount of RAM available for caching database pages across + all connections to the server. The default should be reasonable in most + situations, but for large databases or many simultaneous connections it might + make sense to increase this.

+
+ +
max_wal_size, checkpoint_timeout
+ +
+

These control how often the server forces database pages to be written back + out to the file-system. The defaults are set to minimize disk writes when + ingesting large numbers of records in one session. There should be little + reason to change these values.

+
+ +
ssl_cipher
+ +
+

This controls which ciphers the server allows when negotiating a connection. + The defaults are reasonable, but administrators may want more control. The + setting 'TLSv1.2', for instance, can be used to be compliant with the latest + TLS standard.

+
+
+
+
+ +

The pg_hba.conf file is used to configure which + connections the server accepts for a particular outward facing IP address and what + security mechanisms are enforced for those connections. Currently all addresses are + configured to accept SSL connections only, except possibly for 'localhost'. + Administrators can currently filter connections + based on usernames and the particular database (which corresponds to Ghidra's concept + of repository).

+ +
+

Warning

+ +

By default, the server accepts all connections from all users.

+
+
+ +
+
+
+
+

Configuration Defaults

+
+
+
+ +

There is a serverconfig.xml which contains a few of + the default configuration values that are most crucial for the BSim Database. Beware: This file is currently parsed only once + for the entire lifetime of a particular data + directory: it is read only when the data directory is first initialized, i.e. the first + time the bsim_ctl start command is + invoked on the empty directory. This file is intended to provide reasonable defaults + that are different from the standard PostgreSQL defaults. To provide site specific + configuration, changes should be made to the normal PostgreSQL configuration files.

+
+
+ +
+
+
+
+

Elasticsearch Configuration

+
+
+
+ +

A full description of how to configure an Elasticsearch cluster, including how to + start and stop the server, is beyond the scope of this document. In particular, the bsim_ctl command-line, as described in “PostgreSQL Configuration”, does not apply to + Elasticsearch. Complete documentation is available on-line from the Elasticsearch + website.

+ +
+
+
+
+

Installing the Plug-in

+
+
+
+ +

In order to make use of Elasticsearch with BSim, the database administrator must + install the lsh.zip plug-in as part of the + Elasticsearch deployment. The plug-in is available in the Ghidra add-on named BSimElasticPlugin, which unpacks into a standard + Ghidra installation. The file lsh.zip is a + standard Elasticsearch plug-in that must be installed on every node of the cluster + before a BSim repository can be created. The Elasticsearch distribution typically comes + preconfigured for a single node deployment. The description below shows how to enable + BSim on such a toy deployment, but this will need to be extended to support an entire + cluster.

+ +

Assuming the add-on has been unpacked, the plug-in can be installed to a single node + using the elasticsearch-plugin command in the + bin directory of the node's Elasticsearch + installation.

+ +
+ + + + +
bin/elasticsearch-plugin install + file:///path/to/ghidra/Ghidra/contrib/BSimElasticPlugin/data/lsh.zip
+
+ +

Replace the initial portion of the absolute path in the URL to point to the Ghidra + installation. Once the plug-in is installed, the toy deployment can be (re)started from + the command-line by running

+ +
+ + + + +
bin/elasticsearch
+
+ +

This will dump logging messages to the console, and you should see [lsh] listed among the loaded plug-ins as the node starts + up.

+
+ +
+
+
+
+

The Elasticsearch URL

+
+
+
+ +

Assuming an Elasticsearch cluster is running and the plug-in has been properly + installed, all other parts of BSim interact transparently with the cluster. The bsim command, described in Ingesting + Executables, and the Ghidra/BSim client, described in Querying a BSim + Database, require no additional configuration to work with Elasticsearch, + except users must provide the correct URL to establish a connection. Elasticsearch + communicates over https, and BSim clients + automatically assume they are communicating with Elasticsearch when they see this + protocol. Alternatively, the protocol may be specified as elastic when using the bsim command. Elasticsearch use by BSim assumes a + default port of 9200 unless otherwise specified when specifying the server host. See “Ghidra and BSim + URLs” for additional information about URLs.

+
+
+
+ +
+
+
+
+

Creating a + Database

+
+
+
+ +

If using either PostgreSQL or Elasticsearch the server must be properly configured and + running before a database can be created. In the + case of an H2 file-based database there is no server requirement. Only after a database has + been created can data be ingested or queries performed. In this context, a database is a + single container of reverse engineered functions. Metadata pertaining to executables and + call-graph relationships is also stored, but the principle database record describes a + function. A single PostgreSQL or Elasticsearch + server can hold multiple independent databases.

+ +

A database is created using the bsim + command script. The basic command looks like

+ +
+ + + + +
$(ROOT)/support/bsim createdatabase bsimURL config_template
+
+ +

A BSim database is completely distinct from the Ghidra Server or Ghidra project, so the + executables and functions contained within do not need to coincide at all.

+ +

The Ghidra GUI client specifies a BSim database with its explicit characteristics (i.e., + DB type, name, host/port if applicable, etc.), while the bsim command accepts a bsimURL which includes similar details (see “Ghidra and BSim URLs” for more + details).

+ +

The config_template parameter passed to bsim createdatabase names a collection of specific + configuration values for the newly created database. A standard Ghidra distribution + provides a number of predefined templates (See below) designed for specific database use + cases. It is simplest to use a predefined template when creating a database, but it is + possible to edit an existing template or create a new template (See “Creating Database Templates”).

+ +

There are two critical database properties being determined by the template that need to + be kept in mind: the index tuning and the weighting scheme relative to the size of the database. + The two pieces of the template name, separated by the '_' character, refer to these + concerns.

+ +

The index tuning affects the use of the database by trading off between, the time + required to perform individual queries, the amount of variation between matching functions + a query can tolerate, and the amount of storage required per database record. Ideally, the + database is tuned, before the initial ingest occurs, to the anticipated size of the database. The database can trade off + storage size (per record) and latency for overall query response time, but the decision + needs to be made before the database is populated. Currently there is a medium tuning that is ideal for repositories that will store + on the order of 10 million functions. There is also a large tuning, which uses more storage but can maintain fast + query times for databases with 100 million functions or more. There is a large overlap for + these tunings, so if its unclear how large a database might grow, go ahead and use the + medium tuning.

+ +

The weighting scheme affects how BSim views the relative importance of individual code + constructs within a function. Code constructions are extracted as features, and each feature is assigned a weight. The basic + schemes are: 32 for 32-bit compiled code, 64 for 64-bit code. The scheme that matches the + predominant form of code in the repository being ingested should be used. Mixed schemes are + possible, but a corpus which is predominantly 32-bit, even with a small number of 64-bit + executables mixed in, should still use the 32-bit weights.

+ +

There are some weighting schemes designed for more specialized code. The 64_32 scheme is for 64-bit code using 32-bit pointers. The + nosize scheme allows better matching of 32-bit + functions to 64-bit functions, when they are compiled from the same source. The cpool scheme is designed for Java byte-code or Dalvik + executables. For more discussion of weighting, see “Weighting + Software Features”.

+ +

The full template name incorporates both an index tuning and a weight scheme. Some + common examples of template names:

+ +
+
+
+
medium_32
+ +
+

A medium index tuning with a weighting scheme designed for 32-bit + executables.

+
+ +
medium_64
+ +
+

A medium index tuning with a weighting scheme designed for 64-bit + executables.

+
+ +
large_32
+ +
+

A 32-bit weighting scheme with tuning for a large database size.

+
+ +
medium_cpool
+ +
+

A medium index tuning with a weighting scheme for Java executables.

+
+ +
medium_nosize
+ +
+

A medium index tuning with a weighting scheme allowing matches between 32-bit + and 64-bit code.

+
+
+
+
+
+ +
+
+
+
+

Tailoring BSim + Metadata

+
+
+
+ +

There is some facility to tailor a specific BSim database instance so that it can ingest + and/or report information about executables or functions to make results more useful for a + specific project or user. Capabilities can be added after a database has been created and + is running by issuing specific bsim commands, + but they can also be added to a configuration + template prior to creating the database, which provides a record of the + specific additions should the database instance need to be recreated or multiple tailored + instances be deployed. For additions that allow the ingest of more metadata about + executables or functions, users must provide additional scripts to Ghidra during the ingest + process in order to read in or discover the new metadata.

+ +

The Name, Owner, and Description associated with a BSim instance can be trivially + tailored with the bsim setmetadata + command.

+ +
+ + + + + + + + + + + + +
$(ROOT)/support/bsim setmetadata bsimURL "name=BSim Database"
$(ROOT)/support/bsim setmetadata bsimURL "owner=Administrators"
$(ROOT)/support/bsim setmetadata bsimURL "description=Files of interest"
+
+ +

This information is displayed in various windows by the BSim client. The values can be + changed at any time and do not otherwise affect the records contained in the database. + Multiple command-line parameters can be fed to bsim + setmetadata so long as each one starts with name=, owner=, or + description= respectively. Quoting may be + necessary to get some strings to be interpreted as a single command-line parameter.

+ +
+
+
+
+

Executable Categories

+
+
+
+ +

BSim provides the powerful ability to associate new types of metadata with each + executable that the database ingests. Any method of categorizing executables that + describes an executable with a simple string value, referred to here as an executable + category, can be added as a field to the + database. With only minor adjustments to the ingest process, new category values can be + automatically attached to incoming executables and are treated like any other executable + field that BSim understands. Category values are retrieved with queries, can be used for + filtering, and show up as sortable columns in result tables.

+ +

All categories have a formal name (or type), which is used both in the ingest process + (See below) and as the label for table columns. The name can contain alphanumeric + characters or punctuation from the limited set, ' ._:/()'. For each executable there can + be zero, one, or more string values associated + with the category. No value is required for the executable, and any single value can be + used for filtering (either the executable is labeled with the value or it is not) even if + there are multiple values for that category. If there are multiple values, a query that + matches the executable will return all the values as a single sorted column entry.

+ +

It is also possible to create a special time-based category. This category can have + any name as above, but instead of associating string values with the executable, it + associates a single time-stamp. The time-stamp has precision down to the millisecond and + provides filtering and sorting based on time. Internally, this new category repurposes + the column storage originally providing an executable's Ingest + Date field. As a result, any BSim instance + can have only one time category and only one time-stamp within it. The ingest scripting + must provide any actual time-stamp value for the executable, or the database will fill in + the "epoch", 12:00 am, Jan 1, 1970.

+ +

A new category can be added to the database at any time using the bsim addexecategory command.

+ +
+ + + + + + + + +
$(ROOT)/support/bsim addexecategory bsimURL MyCategoryName
$(ROOT)/support/bsim addexecategory bsimURL MyTimeField date
+
+ +

The single time-stamp field can be renamed by appending the keyword "date" to the + command after the name of the category. Once a category, the corresponding program + options set for any new executables will automatically read into the database as part of + the ingest process. Previously ingested executables, assuming they have the new program + options set, can be updated within the BSim database using one of the bsim updaterepo command variants. In either case, the + relevant program options typically need to be filled by running a Ghidra script (See “Ingesting Executable Categories”). + There is currently no method for deleting a category once it has been created.

+
+ +
+
+
+
+

Function Tags

+
+
+
+ +

BSim can be configured to recognize specific Function + Tags, which are named Boolean properties on individual functions within + an executable. Within a Ghidra program, any number of different function tags can be + established by the user and are used to label individual functions or specific subsets of + functions that share a particular property. This would typically be used to label classes + of functions that are important to analysts, unpacked functions could be labeled with the + tag UNPACKED for instance.

+ +

In order for BSim to recognize specific function tags, they must be individually + registered with the BSim database. These tags are then automatically ingested into the + database, along with the other standard metadata describing functions, and can be used to + filter match results when querying the database. A function tag has a formal name, which + can be displayed as part of the function header within the main code browser and is used + for BSim columns and filter labels. Once the tag is created for a program, functions + universally have the tag as a Boolean property, either the name applies to a function or + it doesn't, and arbitrary subsets can be tagged + with that name.

+ +

A tag must be registered with a BSim database + before it can be used as a filter or seen in results. A tag can be registered at any time + with the bsim addfunctiontag command.

+ +
+ + + + +
$(ROOT)/support/bsim addfunctiontag bsimURL MyTagName
+
+ +

The new tag will automatically be read in when any new executables are ingested. If + previously ingested executables already had the new tags before they were registered, + their metadata within BSim database can be updated using the bsim updaterepo command variants. BSim is limited to 29 + registered tag names, and there is currently no way to remove a tag once it has been + registered.

+
+ +
+
+
+
+

Creating Database Templates

+
+
+
+ +

It is possible to create tailored database configuration templates so that + implementors have a permanent and accessible record of a particular set-up and don't need + to repeatedly issue bsim setmetadata and + bsim addexecategory when creating a + database. Other aspects of a database can also be manipulated, like weighting schemes and + index tuning, but doing this properly is beyond the scope of this document. A database template is the basic set of configuration + parameters used to set up BSim database instance. The configuration parameters are + established for a particular database when the bsim + createdatabase command is run (See “Creating a + Database”). The template name passed on the command-line actually identifies an + XML file-name, appended with the '.xml' suffix, in the directory:

+ +
+ + + + +
$(ROOT)/Ghidra/Features/BSim/data
+
+ +

The file has a root tag of <dbconfig>, + and the first child tag of this root is the <info> tag. This tag contains all the metadata tags that + can be easily changed or added to the database. A list of the metadata tags follows. The + metadata is provided as formal text content within the tag, and none of the tags + currently take attributes.

+ +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
XML TagDescription
<name>Name of the database
<owner>Owner of the database
<description>An overarching description of the database
<major>Major decompiler version used for ingest (Should be set to zero)
<minor>Minor decompiler version used for ingest (Should be set to zero)
<settings>Specific settings for the signature strategy (Should be set to zero)
<execategory>The name of an executable category (to be) defined for this database + instance
<datename>The name of the timestamp column
<functiontag>The name of a function tag (to be) registered with this database + instance
+
+
+ +

There can be multiple <execategory> tags + if more than one category is desired and both <execategory> and <datename> are optional tags. The date column name + defaults to 'Ingest Date' and is drawn from the corresponding Ghidra program option. The + tag order needs to be preserved. There can be multiple <functiontag> tags, one for each function tag to be + registered with the database.

+ +

It is easiest to copy an existing template and just edit the tags described above. The + remaining tags in the file are more dangerous to manipulate. The <k> and <L> + tags pertain to the index tuning. The <weightsfile> tag gives the name of the weights file, + within the same directory, which is also another XML file. It is simplest to choose from + the existing weight files provided with the distribution. See “Weighting Software Features”.

+
+
+
+ + diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/FeatureWeight.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/FeatureWeight.html new file mode 100644 index 0000000000..5eb4a220b5 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/FeatureWeight.html @@ -0,0 +1,258 @@ + + + + + + + + Features and Weights + + + + + + + + + + +
+
+
+
+

Features and Weights

+
+
+
+ +
+
+
+
+

Features of + Software Functions

+
+
+
+ +

The BSim Database uses a standard Feature + Vector approach to compare and index software functions. A feature is an abstraction that simply means a single element + or attribute that can be compared quantitatively between two objects. The set of possible + features used by a particular approach is fixed, and any object being examined is viewed as + some unordered subset of all the possible features. So features are the smallest (atomic) + aspect of an object that can be measured, either two objects share a feature in common or + they do not. But within this scheme, because objects generally consist of many individual + features, quantitative fine-grained comparisons can be automatically calculated.

+ +

The BSim Database extracts its features from the data-flow representation of a function, + after it has been normalized by the Ghidra decompiler. This is the SSA graph representation + of the function, with nodes representing the variables and operators of the function, and + the edges representing the read/write relationships between them. An individual feature is + just a portion of this graph, encompassing some subset of variables and operators and the + specific flow between them. Because of the decompilation, a feature can be viewed naturally + as a uniform snippet of C source code, a partial extraction of some expression in the + source code representation of the function. The full set of features provides uniform (and + overlapping) coverage of the graph representation of the entire function.

+ +

Features encode specific aspects of the variables they cover but not others. The size of + a variable, the operator that produced it, and the set of operators it is fed into are + encoded in the features. But, any name assigned to the variable, its data-type, or even its + storage location are not encoded in the + features.

+ +

Within a function, details about the specific subfunctions that it calls are not encoded + in any of the features for that function, but information describing where the call is made + and the set of parameters it takes is encoded.

+
+ +
+
+
+
+

Weighting + Software Features

+
+
+
+ +

Some features are more useful for identifying a specific function out of a large corpus + than others. With the view that features are just portions of recovered C expressions, some + C expressions are simply more common than others. The BSim Database compensates for these + differences by assigning a weight to each feature that factors in to the similarity and + confidence scores produced when comparing functions. Weighting schemes are considered a + configuration parameter of the database and are established for a particular database when + it is created. The scheme cannot be changed without creating an entirely new database and + reingesting the functions.

+ +

Ghidra comes with precomputed weighting schemes that are calculated using statistics + drawn from homogeneous collections of systems and application software. A feature's weight + is computed by counting the number of times it occurs across the entire corpus and + comparing this with the counts from other features. This allows a direct computation of the + information content of the feature; quantitatively, how much have we narrowed down a + particular function from the corpus when we know it contains a particular feature.

+ +

The two primary weighting schemes are called 32 and 64, based + on 32-bit code and one on 64-bit code respectively. This means that a particular database + instance has better sensitivity for either 32-bit or 64-bit functions. The quantitative + scores, similarity and confidence, will be more accurate at distinguishing pairs of + functions from one corpus. This does not mean that functions from the wrong group cannot be ingested or queried, but the scores may + not be as accurate. There is also a 64_32 + weighting scheme for architectures where code is compiled to use 64-bit registers but + addresses are still 32-bit.

+ +

The specialized weighting scheme nosize + allows BSim to match between 32-bit and 64-bit implementations of a function. It works by + making feature hashes blind to the size difference between a 32-bit variable versus a + 64-bit variable. This compensates for a compiler's tendency to assign a full 64-bit + register to a 32-bit variable, which is frequently difficult for the decompiler to + automatically resolve in the context of a single function. Because of this blindness, there + is a slight loss of sensitivity, when matching 32-bit to 32-bit functions, or when matching + 64-bit to 64-bit, over the 32 or 64 schemes respectively.

+ +

The weighting scheme cpool should be used for + run-time compilation (JIT) architectures, like Java Dalvik or .class byte-code executables. These architectures use + characteristic constant pool instructions that delay + exact decisions about code and data layout until runtime. The decompiler can still recover + data-flow effectively by treating these instructions as black-box operations, so BSim works + in the same way as with native code. But a specialized weighting scheme is needed to + balance BSim's sensitivity to these operations.

+
+ +
+
+
+
+

Comparing Feature + Vectors

+
+
+
+ +

For a particular function, the set of extracted features and their assigned weights make + up the formal feature vector associated with the + function. When querying a BSim Database, the primary function search is performed by + comparing feature vectors. There are two formal scores that are computed on a pair of + feature vectors, similarity and confidence.

+ +
+
+
+
+

Similarity

+
+
+
+ +

Similarity is a direct calculation of the percentage of features in common between two + functions. It varies continuously from 0.0, meaning the functions share no features at + all, to 1.0, meaning that the functions have the same feature set. Formally, similarity + is defined as the cosine similarity of the two + feature vectors. Weights determine how important individual features are in the score + relative to other features, providing a practical and realistic meaning to the score. Two + functions can exhibit a few unimportant changes, but the similarity can still be very + high because the differences are likely not weighted heavily. Along the same lines, two + functions can share most of their features but have a low similarity because they differ + in more important features.

+ +

When searching for a function, the database sets a particular threshold on similarity, + 0.7 by default, and returns functions whose similarity with the queried function exceeds + that threshold. This can produce false positive + matches for small functions because a small function is described by just a few features + and it is then comparatively easy to randomly match a high percentage of these features. + Deciding if a false positive is likely can be decided quantitatively by examining the + confidence score below.

+
+ +
+
+
+
+

Confidence

+
+
+
+ +

Confidence is a log likelihood ratio, a weighted count of the set of features that + match between two functions minus the set of features that are different. It is an + open-ended score, and the higher it gets, the more likely it is that the two functions + are a true match. Fixing a threshold for the confidence score provides a more consistent + false positive rate, as opposed to thresholding on + similarity. A higher score means the two functions have more features in common as an + absolute count, not just a higher percentage. So the chance of randomly matching most of + the features continues to go down as confidence goes up.

+ +

A general correspondence between low confidence scores and false positive rates can be + somewhat skewed by wrappers and other small + functions, which are always common but whose specific frequency can vary depending on the + type of software. BSim fixes the score 10.0 for a particular wrapper form, providing a + convenient boundary between wrappers and more substantial functions where frequencies are + more consistent. For scores of 10.0 and greater, we get the following rough + correspondence with false positive rate. The rate drops by a factor of 2 for an increase + in confidence of between 4 and 5 points.

+ +
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
ConfidenceFalse Positive Rate + (Approximate)
101 in 4,000
261 in 100,000
431 in 1,000,000
931 in 1,000,000,000
+
+
+ +

For a single function, there is an upper-bound to the confidence that can be achieved + by a possible match, its self significance. This + upper-bound is of course reached by comparison with a function having 1.0 similarity. + Self significance is roughly proportional to the size of the function. So its impossible + to achieve a high confidence for a small function, for single matches viewed in + isolation. Of course a medium to low confidence threshold may be enough to produce a + unique match if the database is small, and a medium to high confidence threshold may + still produce occasional false positives if the database is very large.

+
+
+
+ + diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSim/IngestProcess.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/IngestProcess.html new file mode 100644 index 0000000000..e493e53eba --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSim/IngestProcess.html @@ -0,0 +1,732 @@ + + + + + + + + Ingesting Executables + + + + + + + + + + +
+
+
+
+

Ingesting Executables

+
+
+
+ +
+
+
+
+

Ingest + Process

+
+
+
+ +

The process of ingesting binaries into a BSim Database server, in order to exploit + Ghidra's data-flow indexing capabilities, can be subdivided into 4 major aspects.

+ +
+
+
    +
  1. Importing Executables to a Ghidra Server
    2. + +
  2. Analysis
    3. + +
  3. Generating Features and Function Metadata
    4. + +
  4. Importing Features to a BSim Database
    5. +
+
+
+ +

Its possible to populate and use the signature capabilities without putting the binaries + in a Ghidra Server, but it is much easier to generate signatures if step 1 is performed at + some point. Steps 1 and 2 can be performed in either order, or they can be performed + simultaneously. If a Ghidra Server is used, steps 3 and 4 are easily accomplished with the + bsim command-line script.

+ +

In the following examples, we assume that a machine localhost is running both a Ghidra Server and a BSim PostgreSQL database + server. On the Ghidra Server, a repository named repo has + been created. On the BSim server, a database named repo has + also been created. See See Command-Line Utility Reference for more + details on use of bsim command and other supported BSim databases.

+ +
+
+
+
+

Analysis

+
+
+
+ +

Performing Ghidra's normal auto-analysis and ingesting to a Ghidra Server is + accomplished with the analyzeHeadless script, designed + explicitly for this purpose.

+ +
+ + + + +
$(ROOT)/support/analyzeHeadless + ghidra://localhost/repo -import /path/to/rawexes -preScript + TailoredAnalysis.java
+
+ +

This imports all executables in the directory /path/to/rawexes to the repository, running the TailoredAnalysis script first and then performing Ghidra's + auto-analysis. Using these pre and post scripts, there is a great deal of flexibility in + how the executables can be analyzed. For a more complete discussion of the available + options, look at the analyzeHeadlessREADME.txt.

+
+ +
+
+
+
+

Generating Features and Function + Metadata

+
+
+
+ +

To generate features and metadata on an existing repository, use the bsim generatesigs command. Signatures may be written as + XML files to a local directory and/or comitted directly to a specified BSim database. If + not immediately comitting to a database and only storing the XML files an appropriate + database config= may be specified in lieu of a BSim database URL + (bsimURL) if database specific executable categories and function tags are not + utilized. Use of the config= option does not require a running BSim server.

+ +
+ + + + +
$(ROOT)/support/bsim generatesigs + <ghidraURL> </xmldirectory> config=<config_template> + [--overwrite]
+ $(ROOT)/support/bsim generatesigs <ghidraURL> </xmldirectory> + bsim=<bsimURL> [--commit] [--overwrite]
+ $(ROOT)/support/bsim generatesigs <ghidraURL> + bsim=<bsimURL>
+
+ +

Example (generate signature XML files without BSim database commit):

+ +
+ + + + +
$(ROOT)/support/bsim generatesigs + ghidra://localhost/repo/folder /xmldirectory + bsim=postgresql://localhost/repo
+
+ +

Example (generate signature XML files and commit to the BSim database):

+ +
+ + + + +
$(ROOT)/support/bsim generatesigs + ghidra://localhost/repo/folder /xmldirectory bsim=postgresql://localhost/repo + --commit
+
+ +

Special XML files encoding the metadata are written out to the directory /xmldirectory. Every executable indicated by the repository folder + specified by the ghidraURL will have metadata generated, one file per + executable, with a file name derived from the MD5 hash of the original executable. + Repository folders will be traversed recursively. The URL can include a specific path + under the repository, in order to select just a portion of the entire repository for + extraction. Output is primarily per function, indicating the functions name, various + attributes, its feature vector, and the list of other functions it calls.

+ +

A partially completed bsim generatesigs + command can be safely restarted by giving it the same XML directory containing the + partial set of metadata files. Currently it will emit non-fatal warnings for programs in + the repository that were previously processed. You can force it to overwrite the + generated XML files by adding the explicit keyword --overwrite as another parameter.

+ +

Both the Ghidra Server and the BSim server must be running in order for this command + to succeed, as the BSim server provides configuration information that may be relevant to + the signature generation process, such as database specific executable categories or + function tags. Assuming this is the same template used to create the database, the BSim + server no longer needs to be running. As in the example above, configuration information + is pulled from the BSim server and signatures are generated from the Ghidra Server + executables.

+
+ +
+
+
+
+

Importing Features to a BSim + Database

+
+
+
+ +

Importing XML signature files into a BSim database which were previously generated is + done using the bsim commitsigs command.

+ +
+ + + + +
$(ROOT)/support/bsim commitsigs + postgresql://localhost/repo /xmldirectory [override=ghidraURL]
+
+ +

This command takes XML signature files in /xmldirectory + and writes the metadata in them to a BSim database. All the executable, function, and + feature vector records are committed to their appropriate tables and all the indexing is + updated if supported. The URL refers to the BSim database rather than the Ghidra Server. + The URL cannot be extended with a path. Any executable paths are already encoded within + the XML file data.

+ +

Every executable described within the XML files has a repository and path + associated with it in the form of a ghidra:// URL + that was recorded when the XML files were generated. This path can be overridden with the + optional override= parameter where a revised + Ghidra URL may be specified.

+ +

The bsim commitsigs command can be + safely restarted if an initial run is terminated prematurely. Currently, a restart will + emit a non-fatal warning for each executable that was previously committed.

+ +
+

NOTE: The Ghidra Server used to generate the XML metadata files + does not need to be running for this command to succeed. But, the BSim server must be + running.

+
+
+ +
+
+
+
+

Tailoring + Analysis

+
+
+
+ +

It may be necessary as part of the ingest process to alter the way that Ghidra + performs its basic analysis or to add additional steps that gather extra metadata. This + is accomplished by passing a pre-script to the analyzeHeadless command. The script is run once for each executable + analyzed, prior to normal analysis. BSim comes with a sample script TailoredAnalysis.java, such as:

+ +
+
+import ghidra.app.script.GhidraScript;
+import ghidra.framework.options.Options;
+import ghidra.program.model.listing.Program;
+
+public class TailoredAnalysis extends GhidraScript {
+
+        @Override
+        public void run() throws Exception {
+                Options pl = currentProgram.getOptions(Program.ANALYSIS_PROPERTIES);
+                pl.setBoolean("Decompiler Parameter ID", false);
+                pl.setBoolean("Stack", false);
+        }
+}
+   
+
+
+ +

The relevant option group is Program.ANALYSIS_PROPERTIES, which contains program specific + properties for all of Ghidra's analysis passes. The example sets the specific options, + "Decompiler Parameter ID" and "Stack", to false, which causes Ghidra to skip those + passes. Other passes can be toggled or have their parameters altered in this way, see “Analysis Effects on Feature + Extraction” for some of analysis passes that can effect BSim.

+ +
+
+
+
+

Ingesting Executable Categories

+
+
+
+ +

If the BSim database has been tailored to ingest special executable categories, the extra metadata associated with an + executable must be explicitly calculated and stored on the formal Ghidra program in + order for BSim to see it. This is also accomplished with a Ghidra script passed to + analyzeHeadless. BSim expects to find specific + category values stored as Ghidra program options under Program.PROGRAM_INFO. A value can be set by passing the + category name and the value itself as strings to the setString method on the Options object. The following example sets values for the + category "WebResource".

+ +
+
+@Override
+public void run() throws Exception {
+        Options pl = currentProgram.getOptions(Program.PROGRAM_INFO);
+            String value = discoverWebResource();
+            String value1 = discoverAnotherWebResource();
+            Date compiledate = lookupCompileDate();
+            pl.setString("WebResource",value);
+            pl.setString("WebResource_1",value1);
+            pl.setDate("Compile Date",compiledate);
+}
+
+
+ +

The script has all the responsibility for constructing any actual value, and it can + be run either as a pre-script or a post-script. Thus it can use the results of Ghidra's + basic analysis to calculate the value or retrieve it from some external source like a + CSV file. Once the script sets the value, assuming the executable category has already + been added to the BSim database instance, there are no additional steps to take, and + the rest of the BSim ingest process will make sure the value is incorporated into the + database.

+ +

If more than one value needs to be assigned to the same category, the script must + assign the first value to the option matching the category name and then assign + additional options with names obtained by appending '_1', '_2', and so on to the + category name. For the executable time-stamp, BSim reads the option matching the name + of the specific Date executable category. This + can be set within a script by using the setDate + method instead of setString. If no Date category has been created, BSim will read the 'Date + Created' option, which is normally filled in with the time at which Ghidra was created + and analysis started. For additional discussion see “Executable + Categories”.

+
+ +
+
+
+
+

Ingesting Function Tags

+
+
+
+ +

If a set of function tags have been + registered with the BSim database, some form of analysis must still place the tags on + individual functions in a Ghidra program. As with executable categories, this can be + accomplished with a Ghidra script passed to analyzeHeadless. Once a set of functions is identified, the + tag(s) can be added using methods on the standard Function object. This script snippet looks up the function + object by address and then sets the LOGGING tag + on it and removes the SERIALPORT tag.

+ +
+
+public void adjustTags(Address myaddress) throws Exception {
+        Function func = program.getFunctionManager().getFunctionAt(myaddress);
+        func.addTag("LOGGING");
+        func.removeTag("SERIALPORT");
+}
+
+
+ +

If the tag did not exist before the first call to addTag, it will be created. Assuming the tag has been + registered with BSim, it will now automatically be included as part of the metadata of + any functions it was added to. For additional discussion see “Function + Tags”.

+
+
+ +
+
+
+
+

Analysis + Effects on Feature Extraction

+
+
+
+ +

Auto-analysis must be performed in some form, + in order to perform disassembly and identify functions, before features can be generated. + Only code that has been formally identified as a function by Ghidra will have its + features extracted and ingested by the successive steps. The single most important driver + of success for future queries finding an executable is the amount of functions that have + been formally identified by analysis. For most common file formats and architectures, + Ghidra's default settings will provide good coverage, but depending on the type of + executable data, it may be necessary to provide additional scripts to the analysis + process.

+ +

Ghidra has numerous Analyzers, that can be + turned on or off during ingest. Many of these can affect code coverage and should be left + on be default. A small number of settings, on + Analyzers or elsewhere, can actually change which features are extracted for a function, + even though code coverage is unaffected. The rule of thumb is that, among those settings + that can affect features, the settings used during ingest should also be the same + settings used by the end-user when they are analyzing an unknown binary and querying + against the database. In most cases, the best approach is to use the default settings for + both ingest and query, but if you make a change, be sure to make it on both sides. A + difference in settings may not completely prevent successful queries, but is likely to + introduce some amount of noise.

+ +

Analyzers which can affect features include:

+ +
+
+
+
Call-Fixup + Installer
+ +
+

This controls how the decompiler works with prologue, epilogue, and other + special compiler bookkeeping functions. There is little reason to turn this + off.

+
+ +
Decompiler Parameter + ID
+ +
+

This does a global analysis of how individual functions are passing parameters + which may affect the features for some functions, although in most cases the + effect will be small. Currently it's safest to turn this off.

+
+ +
Known Functions That Do No + Return
+ +
+

This identifies certain common functions that do not return to the caller. + This can have a limited effect on features of functions that call these routines. + There is little reason to turn this off.

+
+ +
Shared Return + Calls
+ +
+

This identifies common compiler optimizations where a subfunction is accessed + via a jump instruction rather than a + call. Misidentifying these + instructions can have a big effect on extracted features. There are only a + limited set of circumstances where it makes sense to turn this Analyzer off.

+
+
+
+
+ +

Scripts can be used to alter settings on any number of objects during the analysis + process. A few of these can have significant impact on extracted features. With the + exception of symbol names and data-types, which do + not have an effect on features, anything that changes decompilation will + likely change the extracted features. These settings include:

+ +
+
+
+
Call-Fixup, Inline, No Return
+ +
+

Toggling these settings in a function prototype will affect the features of + any calling function.

+
+ +
Read-only + Settings
+ +
+

Memory blocks or individual data items can be marked as read-only, causing the decompiler to propagate the + underlying value as a constant.

+
+ +
Register + Settings
+ +
+

Its possible to mark registers as holding specific values for specific + functions. The decompiler will respect this and likely propagate the + constant.

+
+
+
+
+
+ +
+
+
+
+

Maintenance

+
+
+
+ +

The BSim server currently has minimal maintenance functionality. Substantial changes + and additions may occur in the near term. It is possible currently to use the SQL command + line tool psql bundled with PostgreSQL in + order to make changes directly to the tables. But for very large modifications to the + database, the best option may be to recreate the database, which is slightly less onerous + than it sounds. The most CPU intensive part of the ingest process, Ghidra's + auto-analysis, typically does not need to be rerun across everything. Regenerating the + metadata files and reimporting takes much less time. Additional efficiency may be gained + by dropping and then regenerating the main index after (re)ingesting. (See below)

+ +
+
+
+
+

Deleting Executables

+
+
+
+ +

The database currently cannot remove individual function records. The records for an + entire executable, and all the functions associated with it, can be removed. To remove + an executable, use one of the following forms of the delete command:

+ +
+ + + + + + + + +
$(ROOT)/support/bsim delete bsimURL md5=7abf...
$(ROOT)/support/bsim delete bsimURL name=...
+
+ +

In the md5 form, you specify the 32 character + hex representation of the md5 hash of the executable, which should identify it + uniquely. Using the name form, there is the + possibility that the name is not unique, in which case the command will fail.

+ +

If a unique executable is identified, its metadata record will be removed, and the + records for all functions which that executable formally contains will also be removed. + For deployments which also maintain a loosely coupled Ghidra Server, keep in mind that + removing executables from the BSim database with these commands does not remove the + corresponding program from the Ghidra Server, this requires an additional step.

+
+ +
+
+
+
+

Updating Names and Other Metadata

+
+
+
+ +

It is feasible to delete an executable from the database and then reingest it in + order to populate updated information into the database, but this is fairly inefficient + because you need to reperform all the function decompilation. Additionally this causes + a large rearrangement of the database tables in order to perform what may amount to a + very small change. The BSim database supports an update operation designed to make in-place changes to the + metadata describing executables and functions. This is primarily useful for + synchronizing function names with the BSim database as analysts rename them and commit + their changes to a Ghidra Server, however other metadata, like executable architecture + or category properties, can also be changed.

+ +

Very similar to the two ingest commands, generatesigs and commitsigs, there are also two update commands generateupdates and commitupdates which are invoked as follows:

+ +
+ + + + +
$(ROOT)/support/bsim generateupdates + <ghidraURL> </xmldirectory> config=<config_template> + [--overwrite]
+ $(ROOT)/support/bsim generateupdates <ghidraURL> </xmldirectory> + bsim=<bsimURL> [--commit] [--overwrite]
+ $(ROOT)/support/bsim generateupdates <ghidraURL> bsim=<bsimURL>
+
+ $(ROOT)/support/bsim commitupdates <bsimURL> + </xmldirectory>
+
+ +

The generateupdates command produces + stripped down metadata XML files for every executable contained within the repository + folder specified by the ghidraURL. Just like the generatesigs command, it can take an optional config=config_template parameter, which + allows the command to execute without the BSim server running. It can also take an + optional --overwrite parameter, causing it + to overwrite any previously generated XML files. If a + bsim=bsimURL is specified with the --commit + option updates will be commited directly to the database. A BSim database commit is + always performed using the specified bsimURL if an xmldirectory is + not specified.

+ +

The commitupdates command commits the + generated metadata to the BSim server. Only the smallest required change is issued as a + formal transaction to the database, and the number of changes that are actually made + are reported per executable. Functions and executables cannot be added or deleted to + the BSim database using this command. If the update file describes new functions in a + preexisting executable, this command will issue warnings about the existence of the new + functions but will not create function records. Other functions will still get + updated.

+
+ +
+
+
+
+

Dropping the Feature Vector Index

+
+
+
+ +

NOTE: Applies to PostgrSQL or Elasticsearch databases only

+ +

For those users performing large ingests or who find themselves rebuilding the + database frequently, it is possible to drop the main index, ingest data, then recreate + the main index. This is likely to be faster overall than the default behavior of + updating the index as data is ingested. To drop the index, use the command:

+ +
+ + + + +
$(ROOT)/support/bsim dropindex bsimURL
+
+ +

Once the data has been ingested, rebuild the index with the command:

+ +
+ + + + +
$(ROOT)/support/bsim rebuildindex bsimURL
+
+ +

The time it takes to rebuild depends directly on the number of functions that have + been ingested. For very large collections, rebuilding can take hours or days. The + database can still be accessed while the index is dropped, but query times will likely + be too long for the database to be usable.

+
+ +
+
+
+
+

Prewarming the Database

+
+
+
+ +

NOTE: Applies to PostgrSQL databases only

+ +

A maintainer can issue the bsim + prewarm command to prepopulate RAM with commonly accessed portions of a + BSim database.

+ +
+ + + + +
$(ROOT)/support/bsim prewarm + ghidra://localhost/repo
+
+ +

Without this command, initial queries to a cold database (one that has just been restarted) can run + slowly, giving poor response times until the cache is fully populated. A query + typically needs to access an effectively random subset of pages making it a bad method + for refreshing the cache.

+ +

The command is intended to be run once, immediately after restarting a server and + before any queries have been made. It attempts to quickly preload the main vector + index, and possibly portions of other tables, into RAM, by reading from disk + sequentially. Queries may continue to improve as the database optimizes its cache + across all tables, but the command effectively eliminates slow initial queries.

+
+
+ +
+
+
+
+

Migration

+
+
+
+ +

BSim is a prototype capability, and the database layout may be subject to change. We + intend to minimize the impact of this to the extent possible, and in particular, major + changes should be limited to major Ghidra releases. But its possible in general that an + existing BSim database will be incompatible with both the client and server from a new + release.

+ +

Unfortunately, the only option to upgrade in these case is to reingest the executables + into a new BSim database. Frequently the first two stages of ingest (See Ingesting + Executables), importing executables to a Ghidra Server and running auto-analysis, + do not need to be repeated. Only the final two stages need to be performed, generating + and importing features to the BSim server, usually accomplished with the generatesigs and commitsigs commands.

+ +

BSim fundamentally depends on the Ghidra decompiler, which steadily adds new analysis + features that can affect compatibility over time. Many additions have no effect on BSim, + have a small effect, or affect only a tiny percentage of functions. To minimize the + impact to existing databases, the decompiler, independent of the Ghidra release, is + assigned a major and minor version number. A change in the minor number of 1 or 2 + should have little to no impact for most queries, but users will have to tolerate this + rare degradation of results if they place queries using a client that doesn't match the + BSim server's version. If the client and server differ by a major version, queries will + return an error message.

+
+
+
+ + diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/BSimSearch.html b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/BSimSearch.html new file mode 100644 index 0000000000..bdbfd102af --- /dev/null +++ b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/BSimSearch.html @@ -0,0 +1,517 @@ + + + + + + + BSim Search + + + + + +

BSim Search

+ +

The BSim Search features allows the user to perform similar function searches against an + existing BSim database. See the BSim + Overview for a full description of BSim. This section describes the actions and related + GUIs for conducting either an overview search or a similar functions search and the follow on + actions that can be performed on the search results.

+ +

Enabling the BSim Search Plugin

+ +

The BSim Database feature comes with an Ghidra GUI interface for initiating searches. This + GUI integrates with Ghidra via a plug-in that can be added to the main Ghidra Code Browser + tool. This plug-in is currently called the BSimSearchPlugin within Ghidra and can be + enabled from within the Configure Toolmenu. If the plug-in is enabled, the Code Browser + will contain a BSim menu with actions for managing BSim database definitions, performing an + overview query, or perfrorming a similar functions search. Also, the   will appear in the toolbar, which is a shortcut for + search similar fuctions action.

+ +

Defining and Managing BSim Databases

+ +
+

Before a BSim overview or search can be performed, one or more BSim database + specifications must be defined. BSim database specifications are managed by the BSim Server + Manager dialog.

+ +

The server dialog can be invoked either from main toolbar, BSim->Manage Servers, or using the button in either the BSim Overview or BSim Search + dialogs.

+
+
+ + +
+ +

+
+
+ + +

The dialog displays a table showing all the currently defined BSim databases/servers. Each + entry shows a name for the BSim database, its type (postgres, elastic, or file), a host ip + and port (if applicable), and finally the number of active connections.

+ +

There are three primary actions for this dialog:

+ + +
    +
  •  Add a new database/server definition - a + Define Server Dialog will be shown.
    • + +
  •  Delete a database/server definition - The + selected entry will be deleted.
    • + +
  •  Change password - A change password + dialog will appear for the selected entry
    • +
+ +

Defining a new BSim server/database

+ +
+

Pressing the brings up the Add BSim Server + dialog

+
+ + +
+ +

+ + +

Choose the type of BSim database first as that will affect the type of information that + needs to be specified. For postgres and elastic, you need to enter host and port. For file, + you see a button for using a filechooser to pick the file that is the local BSim H2 + database.

+
+
+ +

BSim Overview Query

+ +

The BSim Overview action does a search against all functions in the current program, but + instead of returning all the specific results for each function, it returns a count of the + matching results for each function.

+ +

To invoke an overview search dialog, select BSim+Overview...

+
+ + +
+ +

+
+
+ + +

To start the overview task, select a predefined BSim database server from the combo box or + press the button to bring up the Manage BSim Servers dialog. Then adjust the similarity and + confidence settings as desired, and press the Overview button. See the settings for the Similar Functions Search for more information about similarity and + confidence values.

+ +

BSim Overview Results

+ +

After performing a BSim Overview Query, the BSim Function Overview window will + appear.

+
+ + +
+ +

+ + +

The table displays an entry for each function that had at least one hit. Each entry displays + the address of the function, its name, the number of hits BSim found and the self significance + of the function

+ + +

The action pops up a dialog displaying the search + criteria used to generate this overview results set.

+ +

The action controls whether or not + single clicking in the table will navigate the listing to the selected function. Even if the + action is off, navigate will still occur on a double-click.

+ +

The action will create a selection in the + listing for the selected row(s).

+ +
+

There are also several pop-up actions that work on the selected rows.

+ + +
+ +

When trying to use the Executable Match Summary to determine if there is a significant match + between the currently active program and executables in the database, there are two potential + sources of noise in the resulting scores. Very small functions can produce false positives + which artificially increase the confidence score for a particular executable. Also, some + functions, like those provided by standard language libraries, may be used by a large portion + of the executables in the corpus, and incorporating their matches into confidence scores + obscures more significant matches. In both cases, the functions have an overly large number of + matches in the database. This table, preferably sorted on the Hit Count column, serves as an + overview of what, within the context of the active program, are the most common and least + common functions in the database. This makes it easy to filter out precisely these kinds of + problem functions.

+ +

The standard procedure is to select an upper-bound for a function's Hit Count, select every + row in the table below that threshold, and then transfer that selection to the main Code + Browser window by clicking on the Make a selection icon in the upper right corner of the table. + Then, with selection active, invoke the Search Similar + Functions.

+ +

BSim Similar Function Search

+ +

The BSim Similar Function Search action performs a BSim search against one or more functions + in the current program. If there is a selection, it will search all functions in the selection; + otherwise it will search on the one function containing the cursor (If the cursor is not in the + body of any function, an error dialog will appear).

+ +

To invoke an overview search dialog, select BSim+Search Functions... or press the   button in the main toolbar.

+
+ + +
+ +

+
+ + +

This dialog allows you to configure the BSim search. The fields are as follows:

+ +
+

Standard Fields

+ +
    +
  • BSim Server - Choose a BSim database/sever from the + drop down list or define a new one using the + button.
    • + +
  • Function(s) - Shows the function to be searched + or the number of functions being searched. Press the button to see a list of all the selected functions.
    • + +
  • Similarity Threshold - This is a score between 0.0 + and 1.0 indicating as a percentage how similar two functions are to each other and is + calculated as a Cosine Similarity on their feature sets. For two functions that are about + the same size, a similarity score of 0.85 indicates that they share about 85% of their + features. The score takes into account the relative importance of individual features and + will vary from a raw feature percentage.
    • + +
  • Confidence Threshold - Enter a minimum confidence + threshold for matches. Confidence is an unbounded score that indicates how likely it is for + a given pair of functions to be a causal match. Formally, confidence is a log likelihood + ratio that accumulates positive scores for features in common and negative scores for + differences. The larger the score, the more significant the result.
    • + +
  • Max Matches Per Function - Enter the maximum number of + results to return for any one function
    • +
+ +

Filters

+ +

Filters allow the search to be further restricted by allowing the user to choose from a + list of predefined filter criteria and then specify a value for that critera. Supported + filters include:<\P>

+ +
    +
  • Executable Name - specify one or more executable names. Only functions from + executables with those names will be included.
    • + +
  • MD5 - specify a 32 Hex digit MD5 string. Only functions from executables with + that MD5 will be included.
    • + +
  • Architecture - Specify one or more Ghidra architecture specifications. Only + functions from that type of executable will be included.
    • + +
  • Compiler - Specify one or more Ghidra compiler specifications. Only functions + with those compiler specifications will be included
    • + +
  • Path Starts With - Specify one or more path strings. Only functions from + executables with names that start with that path are included.
    • + +
  • Calls - Specify one or function names. Only functions that calls functions with + those names will be included.
    • + +
  • Ingest Date Earlier - specify a date. Only executables that were ingested into + Ghidra prior to that date are included.
    • + +
  • Ingest Date Later - specify a date. Only executables that were ingested into + Ghidra after that date are included.
    • + +
  • Function tagged as <TAG> - if true, only include functions that were + tagged with <TAG> where <TAG> is some predefined function tag string. If false, + only include functions that don't have that tag association.
    • +
+ +

Most of these filters also have NOT versions where only functions that don't match the + criteria are included in the results.

+
+ +

Once all the fields have valid values, press the Search button to initiate the BSim + Function search.

+ + +

A BSim search can also be initiated from + either the listing or decompiler by right-clicking to bring up the popup menu and selecting + either BSim->Search Function(s) or + BSim->Search Function(s)... The only + difference is whether or not to bring up the BSim Search Dialog before performing the search. + Once one search has been done, subsequent searches can be done using the same settings as the + previous search without bringing up the dialog.

+ +

When this action is invoked from the listing, it will apply the function containing the + cursors, unless there is a selection, in which case it applies to all functions in the + selection. When the action is invoked from the decompiler, it will apply to the function whose + name is directly under the cursor.

+ +

Similar Function Search Results

+ +

After initiating a BSim Similar functions search, a BSim Search Results Window will + appear.

+ +
+ +

+ + +

There are two panels associated with each result set. The top panel is the Function Matches + table and the bottom table is the Executables Summary Table. The Executables Summary Table can + be hidden using the tool bar button. Each row will + show columns pertaining to the particular match including, the name of the original function + queried, the name of the matching function, and the corresponding similarity score. If a single + function produces more than one match, each match will produce a separate row. Clicking on the + column headers will sort the results on that column, and clicking on individual rows will + navigate in the Code Browser to the original function that produced that particular match.

+ +
+

Function Matches Panel

+ +

The Function Matches Panel displays one function match result per row. There can be + multiple rows/matches for any queried function. Each row displays the name of the function + being queried, the name of the matching function, its associated match scores, and other + related information described below. Clicking on column headers will sort on that column and + clicking on a row will navigate the tool to function being queried. The columns include (not + all are visible by default):

+ +
    +
  • Status - This is the status result of attempting to apply names from a matching + function to the queried function. Hover on the status to get a description of the apply + status.
    • + +
  • Architecture - The standard Ghidra language ID specifying the + processor/architecture of the matching function.
    • + +
  • Category - A user controlled field for labeling groups of executables.
    • + +
  • Compiler - The compiler used to build the matching function/executable.
    • + +
  • Confidence - The confidence score associated with the match.
    • + +
  • Exe Name - The name of the executable containing the matching function.
    • + +
  • Ingest Date - The date and time when the executable containing the matching + function was ingested into the database.
    • + +
  • Location - The address of the function being queried.
    • + +
  • Matching Function Name - The name of the matching function.
    • + +
  • Md5 - The MD5 checksum of the executable containing the function match.
    • + +
  • Query Function Name - The name of the function being queried.
    • + +
  • Similarity - The similarity score associated with the match.
    • + +
  • Matches - The number of matches found for the source function.
    • + +
  • KNOWN_LIBRARY - Boolean value (check-box) indicating the function is a known + statically linked library function, as labeled by the FunctionID (FID) analyzer.
    • + +
  • HAS_UNIMPLEMENTED - Boolean value (check-box) indicating the function contains + instructions that weren't recognized by the decompiler.
    • + +
  • HAS_BADDATA - Boolean value (check-box) indicating that decompilation of the + function was in error or incomplete.
    • +
+ +

Executable categories added to the specific database instance will also be available here + as additional columns. See Executable + Categories. The column name will match the formal category name, and the string values + can be sorted like any other column. It is possible for multiple values to be assigned to the + same category for a single executable. In this case, the results table will still display a + single column, but the cell will display all the values as a sorted and comma-separated list. + If a new date column is specifically added to the database, this will replace the existing column called 'Ingest Date'. In either case, + this column will sort and filter as a proper date.

+ +

Each function tag registered with the BSim instance will produce an additional column + available here. See Function + Tags. The column will be labeled with the tag name, and the row entry will be a + check-box, indicating whether the tag was present for that function or not.

+ +

Executable Match Summary

+ +

The Executable March Summary table displays a row for each executable that has at least + one matching function for the queried function(s). Every function returned as a match is + associated with its own executable. This table lists exactly one row for every executable + associated with some function match, even if there is more than one such match. Many of the + columns are the same as for the Function Match table, but there are two columns that show an + aggregated value over all function matches that share that same executable.

+ +
    +
  • Confidence - For a query taken from functions in an unknown executable, + the greater the number of function matches that come from a single executable, the more + significant that executable is overall as a match to the unknown executable. The per + executable confidence quantifies this by summing all the function match confidence + scores into a single executable score.
    • + +
  • Function Count - A simple count of the number of function matches that share + this same executable.
    • +
+ +

The executables with the highest aggregate confidence scores share the highest + amount of functionality with the subset of functions in the active program that were queried. + Users should be aware that this shared functionality is not necessarily the most important + functionality. Small functions can produce false positive matches that artificially inflate a + confidence score, and matches to library functions increase the score even though the shared + functionality is not significant. Proper filtering of the queried subset and of the results + may be crucial to getting a meaningful result. See The + Overview Query.

+ +

Actions

+ +

Toolbar Actions

+ +
    +
  •   Search + Info- Pops up a dialog displaying the search criteria used to generate this results + set.
    • + +
  •   + Searched Functions- Pops up a dialog showing a table of all searched functions and + the match count for each.
    • + +
  •  Post Filters - Pops up a dialog for creating + post search filters to further reduce the data being displayed in the table. See BSim Filters for a description of the filters.
    • + +
  •   Hide/Show Executables Summary Table- Toggles the + Executables Summary table on or off.
    • + +
  • > Navigate on table selection- Toggles + whether or not single clicking in the table will cause the tool to navigate. Double + clicking will always navigate.
    • +
+ +

Popup Actions on Functions Table

+ +
    +
  • Apply Function Name - For each selected row, Ghidra + will attempt to apply the name and namespace of the matching function to the queried + function. If more than one match has been selected for the queried function, then nothing + will be applied.
    • + +
  • Apply Function Signature - For each selected row, + Ghidra will attempt to apply the name, namespace and skeleton signature of the matching + function to the queried function. A skeleton signature is a signature where all + non-primitive data types have been replaced with empty placeholder structures. This is safer + than using all the data types which may not be appropriate for the target program as BSim + finds matches against programs with differing architectures and compilers. If more than one + match has been selected for the queried function, then nothing will be applied.
    • + +
  • Apply Function Signature and + Data types - For each selected row, Ghidra will attempt to apply the name, namespace, + signature,and referenced data types of the matching function to the queried function. This + action should be used with caution as it could result in using many data types that are + not appropriate for the target function, especially when applying signatures from different + architectures and compilers. If more than one match has been selected for the queried + function, then nothing will be applied.
    • + +
  • Clear Error Status - Clears any apply error + statuses on the selected row(s).
    • + +
  • Compare Functions - The Function Comparison + window will be displayed for the selected query and matching function.
    • +
+ +

Popup Actions on Executables Summary Table

+ +
    +
  • Filter on this Executable - This will cause a + filter to be applied to the Function Matches table such that only matches from the selected + Executable will be displayed.
    • + +
  • Load Executable - The selected program will be + loaded and displayed in the active tool.
    • +
+ +

Comparing Functions

+ +

For an interesting function match, the user can invoke the CodeDiffPlugin in order + to display the decompilation of the two matching functions side-by-side and highlight the + differences between them. In order for this to happen, the tool needs to have access to the + matching executable. The executable can be pulled in automatically if the Ghidra server + corresponding to the BSim Database is running. Every executable record in the database has a + URL field providing the host, repository, and path for retrieval. If the executable records + were ingested from a Ghidra server using the standard tools, this field should be populated + correctly.

+ +

The code comparison is triggered by right-clicking on a particular entry in the Function + Match table and selecting Compare Function from the resulting pop-up. If the Ghidra + server containing the executable is running, it will be loaded as a separate program directly + into the current Code Browser and a comparison window will be displayed. If the Ghidra server + is not running, or if the URL field is missing from the record, the comparison will still be + triggered if the matching executable is loaded manually as a program in the same Code + Browser. The menu action will identify the executable by name.

+ +

Loading Executables

+ +

An executable can be loaded into the Code Browser, without immediately triggering a + function comparison, by right-clicking on a row in the Executables Summary table and + selecting Load Executable from the corresponding pop-up menu.

+
+ +

Authentication

+ +

Depending on the configuration of the database (See Security and + Authentication), the user may need to authenticate themselves with the BSim server. This + check will be performed immediately upon selected a server definition. If the server requires a + password, a separate dialog will be brought up.

+ +

By default Ghidra will connect as the username reported by the OS, but in the password + dialog, a different username can be entered if this doesn't match the account established on + the server. The title bar of the main dialog indicates the username being used for the current + connection.

+ +

If the BSim server requires PKI authentication, the user must register their certificate + with the Ghidra client. This is accomplished from the main Project window by selecting Set + PKI Certificate... from the Edit menu and pointing the dialog at the certificate + file. The same certificate is used for authenticating with BSim and with a Ghidra server, if + either require PKI. Ghidra will typically bring up a password dialog once per session to unlock + the certificate at the first point it is required.

+ + diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/AddServerDialog.png b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/AddServerDialog.png new file mode 100644 index 0000000000..1210541490 Binary files /dev/null and b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/AddServerDialog.png differ diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/ApplyResultsPanel.png b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/ApplyResultsPanel.png new file mode 100644 index 0000000000..c32fbf5211 Binary files /dev/null and b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/ApplyResultsPanel.png differ diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewDialog.png b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewDialog.png new file mode 100644 index 0000000000..b10876f8cc Binary files /dev/null and b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewDialog.png differ diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewResults.png b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewResults.png new file mode 100644 index 0000000000..b517fb6915 Binary files /dev/null and b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimOverviewResults.png differ diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimResultsProvider.png b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimResultsProvider.png new file mode 100644 index 0000000000..236b6d5756 Binary files /dev/null and b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimResultsProvider.png differ diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimSearchDialog.png b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimSearchDialog.png new file mode 100644 index 0000000000..3b9f63d69e Binary files /dev/null and b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/BSimSearchDialog.png differ diff --git a/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/ManageServersDialog.png b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/ManageServersDialog.png new file mode 100644 index 0000000000..abe6af8494 Binary files /dev/null and b/Ghidra/Features/BSim/src/main/help/help/topics/BSimSearchPlugin/images/ManageServersDialog.png differ diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/BSimSearchPlugin.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/BSimSearchPlugin.java new file mode 100644 index 0000000000..a92c9350f0 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/BSimSearchPlugin.java @@ -0,0 +1,624 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui; + +import java.util.*; + +import javax.swing.Icon; +import javax.swing.SwingUtilities; + +import docking.DockingWindowManager; +import docking.action.builder.ActionBuilder; +import docking.widgets.OkDialog; +import generic.lsh.vector.LSHVectorFactory; +import generic.theme.GIcon; +import ghidra.app.context.ListingActionContext; +import ghidra.app.decompiler.ClangFuncNameToken; +import ghidra.app.decompiler.ClangToken; +import ghidra.app.plugin.ProgramPlugin; +import ghidra.app.plugin.core.decompile.DecompilerActionContext; +import ghidra.features.bsim.gui.overview.BSimOverviewProvider; +import ghidra.features.bsim.gui.search.dialog.*; +import ghidra.features.bsim.gui.search.results.BSimSearchResultsProvider; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.BsimPluginPackage; +import ghidra.features.bsim.query.FunctionDatabase.ErrorCategory; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.facade.*; +import ghidra.features.bsim.query.protocol.*; +import ghidra.framework.model.DomainObject; +import ghidra.framework.plugintool.PluginInfo; +import ghidra.framework.plugintool.PluginTool; +import ghidra.framework.plugintool.util.PluginStatus; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.*; +import ghidra.program.model.pcode.PcodeOp; +import ghidra.program.util.ProgramLocation; +import ghidra.program.util.ProgramSelection; +import ghidra.util.*; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.*; + +/** + * Plugin for BSim search features + */ +//@formatter:off +@PluginInfo(category = "BSim", + description = "This plugin allows users to search selected functions against a database " + + "of previously analyzed functions and returns a table of similar functions", + packageName = BsimPluginPackage.NAME, + shortDescription = "Search Bsim database(s) for similar functions", + status = PluginStatus.STABLE) +//@formatter:on +public class BSimSearchPlugin extends ProgramPlugin { + + static final Icon ICON = new GIcon("icon.bsim.query.dialog.provider"); + public static final String HELP_TOPIC = "BSimSearchPlugin"; + private SFQueryServiceFactory queryServiceFactory = new DefaultSFQueryServiceFactory(); + private Set searchResultsProviders = new HashSet<>(); + private Set overviewProviders = new HashSet<>(); + + private BSimServerManager serverManager = new BSimServerManager(); + + private BSimSearchService searchService; + private BSimServerCache lastUsedServerCache = null; + private BSimSearchSettings lastUsedSettings = new BSimSearchSettings(); + private volatile AbstractProgramTask currentTask; + + private TaskListener taskListener = new TaskListener() { + + @Override + public void taskCompleted(Task task) { + currentTask = null; + } + + @Override + public void taskCancelled(Task task) { + currentTask = null; + } + + }; + + public BSimSearchPlugin(PluginTool plugintool) { + super(plugintool); + searchService = new MyBSimSearchService(); + } + + @Override + protected void init() { + createActions(); + } + + private void createActions() { + new ActionBuilder("BSim Overview", getName()).menuPath("BSim", "Perform Overview...") + .helpLocation(new HelpLocation(getName(), "BSim_Overview_Dialog")) + .onAction(c -> showOverviewDialog()) + .buildAndInstall(tool); + + new ActionBuilder("BSim Search Functions", getName()) + .menuPath("BSim", "Search Functions...") + .menuIcon(ICON) + .toolBarIcon(ICON) + .toolBarGroup("View", "Bsim") + .helpLocation(new HelpLocation(getName(), "BSim_Search_Dialog")) + .onAction(c -> showSearchDialog(getSelectedFunctions())) + .buildAndInstall(tool); + + new ActionBuilder("Manage BSim Servers", getName()).menuPath("BSim", "Manage Servers") + .helpLocation(new HelpLocation(getName(), "BSim_Servers_Dialog")) + .onAction(c -> manageServers()) + .buildAndInstall(tool); + + tool.setMenuGroup(new String[] { "BSim" }, "ZBSIM"); + new ActionBuilder("BSim Search From Listing", getName()) + .popupMenuPath("BSim", "Search Function(s)") + .popupMenuGroup("ZZZ") + .helpLocation(new HelpLocation(getName(), "BSim_Quick_Search")) + .withContext(ListingActionContext.class) + .enabledWhen(c -> lastUsedServerCache != null && canSearchFunctionsInListing(c)) + .onAction(c -> searchService.search(lastUsedServerCache, lastUsedSettings, + getSelectedFunctions(c))) + .buildAndInstall(tool); + + new ActionBuilder("BSim Search From Listing With Dialog", getName()) + .popupMenuPath("BSim", "Search Function(s)...") + .popupMenuGroup("ZZZ") + .helpLocation(new HelpLocation(getName(), "BSim_Quick_Search")) + .withContext(ListingActionContext.class) + .enabledWhen(c -> canSearchFunctionsInListing(c)) + .onAction(c -> showSearchDialog(getSelectedFunctions(c))) + .buildAndInstall(tool); + + new ActionBuilder("BSim Search From Decompiler", getName()) + .popupMenuPath("BSim", "Search Function") + .popupMenuGroup("ZZZ") + .helpLocation(new HelpLocation(getName(), "BSim_Quick_Search")) + .withContext(DecompilerActionContext.class) + .enabledWhen(c -> lastUsedServerCache != null && isDecompilerOnFunction(c)) + .onAction(c -> searchService.search(lastUsedServerCache, lastUsedSettings, + getFunctions(c))) + .buildAndInstall(tool); + + new ActionBuilder("BSim Search From Decompiler", getName()) + .popupMenuPath("BSim", "Search Function...") + .popupMenuGroup("ZZZ") + .helpLocation(new HelpLocation(getName(), "BSim_Quick_Search")) + .withContext(DecompilerActionContext.class) + .enabledWhen(c -> isDecompilerOnFunction(c)) + .onAction(c -> showSearchDialog(getFunctions(c))) + .buildAndInstall(tool); + + } + + public void doBSimSearch(Program program, List
functionAddresses, boolean showDialog) { + Set functions = getFunctions(program, functionAddresses); + if (showDialog) { + showSearchDialog(functions); + } + else { + searchService.search(lastUsedServerCache, lastUsedSettings, functions); + } + } + + private Set getFunctions(Program program, List
functionAddresses) { + Set functions = new HashSet(); + FunctionManager functionManager = program.getFunctionManager(); + for (Address address : functionAddresses) { + Function f = functionManager.getFunctionAt(address); + if (f != null) { + functions.add((FunctionSymbol) f.getSymbol()); + } + } + return functions; + } + + private boolean isDecompilerOnFunction(DecompilerActionContext c) { + if (c.isDecompiling()) { + return false; + } + ClangToken token = c.getTokenAtCursor(); + return token instanceof ClangFuncNameToken; + } + + private boolean canSearchFunctionsInListing(ListingActionContext c) { + return canSearchFunctionsInListing(c.getProgram(), c.getSelection(), c.getLocation()); + } + + private boolean canSearchFunctionsInListing(Program program, ProgramSelection selection, + ProgramLocation loc) { + if (program == null) { + return false; + } + if (selection != null && !selection.isEmpty() && selection.contains(loc.getAddress())) { + return hasAtLeastOneFunctionInSelection(program, selection); + } + return program.getFunctionManager().isInFunction(loc.getAddress()); + + } + + private boolean hasAtLeastOneFunctionInSelection(Program program, ProgramSelection selection) { + FunctionManager functionManager = program.getFunctionManager(); + FunctionIterator iterator = functionManager.getFunctionsNoStubs(selection, true); + return iterator.hasNext(); + } + + private Set getFunctions(DecompilerActionContext c) { + ClangFuncNameToken funcToken = (ClangFuncNameToken) c.getTokenAtCursor(); + PcodeOp op = funcToken.getPcodeOp(); + Address calledAddress = null; + if (op == null || (op.getOpcode() != PcodeOp.CALL)) { + //op is null - user clicked on function token in function signature at the top of the + //decompiler window op.getOpcode != PcodeOp.CALL shouldn't happen but provide + //reasonable default just to be safe + //in either case just query the function currently in the decompiler window + calledAddress = c.getAddress(); + } + else { + calledAddress = op.getInput(0).getAddress(); + } + Program p = c.getProgram(); + Function function = p.getFunctionManager().getFunctionContaining(calledAddress); + return Set.of((FunctionSymbol) function.getSymbol()); + } + + @Override + public void dispose() { + closeAllProviders(); + } + + @Override + protected boolean canCloseDomainObject(DomainObject dObj) { + AbstractProgramTask task = currentTask; + return task == null || task.getProgram().equals(dObj); + } + + @Override + protected void programClosed(Program program) { + // Close any overview or search providers based on that program + + // copy to avoid concurrent mod exception as closing will trigger a callback to remove + List searches = new ArrayList<>(searchResultsProviders); + for (BSimSearchResultsProvider provider : searches) { + if (provider.getProgram() == program) { + provider.closeComponent(); + } + } + + List overviews = new ArrayList<>(overviewProviders); + for (BSimOverviewProvider provider : overviews) { + if (provider.getProgram() == program) { + provider.closeComponent(); + } + } + } + + private void manageServers() { + BSimServerDialog dialog = new BSimServerDialog(getTool(), serverManager); + DockingWindowManager.showDialog(dialog); + } + + private void showSearchDialog(Set functions) { + if (checkBusy()) { + return; + } + if (functions.isEmpty()) { + OkDialog.showError("No Function(s) Selected", + "You must first place your cursor in a function or \n" + + "create a selection that contains 1 or more functions!"); + return; + } + BSimSearchDialog searchDialog = + new BSimSearchDialog(tool, searchService, serverManager, functions); + tool.showDialog(searchDialog); + } + + private boolean checkBusy() { + if (currentTask != null) { + OkDialog.showInfo("BSim Database Busy", "Only one BSim query permitted at a time!"); + return true; + } + return false; + } + + private void showOverviewDialog() { + if (checkBusy()) { + return; + } + BSimOverviewDialog overviewDialog = + new BSimOverviewDialog(tool, searchService, serverManager); + tool.showDialog(overviewDialog); + } + + private Set getSelectedFunctions() { + return getSelectedFunctions(currentProgram, currentSelection, currentLocation); + } + + private Set getSelectedFunctions(ListingActionContext c) { + return getSelectedFunctions(c.getProgram(), c.getSelection(), c.getLocation()); + } + + private Set getSelectedFunctions(Program program, ProgramSelection selection, + ProgramLocation location) { + FunctionManager functionManager = program.getFunctionManager(); + Set functions = new HashSet<>(); + if (selection == null || selection.isEmpty()) { + if (currentLocation == null) { + // must be opening + return functions; + } + + Function currentFunction = functionManager.getFunctionContaining(location.getAddress()); + if (currentFunction != null) { + functions.add((FunctionSymbol) currentFunction.getSymbol()); + } + return functions; + } + + FunctionIterator iterator = functionManager.getFunctionsNoStubs(selection, true); + for (Function function : iterator) { + functions.add((FunctionSymbol) function.getSymbol()); + } + return functions; + } + + public void closeAllProviders() { + for (BSimSearchResultsProvider provider : searchResultsProviders) { + provider.closeComponent(); + } + for (BSimOverviewProvider provider : overviewProviders) { + provider.closeComponent(); + } + } + + /** + * Get all non-stub functions for computing signature overview + * @return set of all non-stub function symbols + */ + private Set getOverviewFunctions() { + Program program = getCurrentProgram(); + if (program == null) { + return Set.of(); + } + + FunctionManager functionManager = program.getFunctionManager(); + TreeSet functions = new TreeSet<>(new Comparator() { + @Override + public int compare(FunctionSymbol f1, FunctionSymbol f2) { + return f1.getAddress().compareTo(f2.getAddress()); + } + }); + + FunctionIterator iterator = functionManager.getFunctionsNoStubs(true); + for (Function function : iterator) { + functions.add((FunctionSymbol) function.getSymbol()); + } + return functions; + } + + private BSimOverviewProvider getOverviewProvider(BSimServerInfo serverInfo, Program program, + LSHVectorFactory vectoryFactory, BSimSearchSettings settings) { + // this call is made from a task, but new providers need to be created in the Swing thread + return Swing.runNow( + () -> new BSimOverviewProvider(this, serverInfo, program, vectoryFactory, settings)); + } + + private BSimSearchResultsProvider getSearchResultsProvider(BSimServerInfo serverInfo, + DatabaseInformation dbInfo, LSHVectorFactory vectorFactory, SFQueryInfo queryInfo, + BSimSearchSettings settings) { + // this call is made from a task, but new providers need to be created in the Swing thread + return Swing.runNow(() -> new BSimSearchResultsProvider(this, tool, serverInfo, dbInfo, + vectorFactory, queryInfo, settings)); + } + + /** + * Get an {@link AutoCloseable} {@link SimilarFunctionQueryService} instance which will + * be connected to current BSim Server. Caller is responsible for closing instance. + * @param program program containing functions to be searched + * @return new {@link SimilarFunctionQueryService} instance + * @throws QueryDatabaseException if error occurs connecting to database + */ + private SimilarFunctionQueryService getQueryService(Program program, BSimServerInfo serverInfo) + throws QueryDatabaseException { + + SimilarFunctionQueryService queryService = + queryServiceFactory.createSFQueryService(program); + queryService.initializeDatabase(serverInfo.toURLString()); + String errorMessage = queryService.getDatabaseCompatibility(); + if ((queryService.getLastError() != null) && + (queryService.getLastError().category == ErrorCategory.Nodatabase)) { + errorMessage = "Database does not exist"; + } + if (errorMessage != null) { + throw new QueryDatabaseException(errorMessage); + } + return queryService; + } + + public void providerClosed(BSimSearchResultsProvider provider) { + searchResultsProviders.remove(provider); + } + + public void providerClosed(BSimOverviewProvider overviewProvider) { + overviewProviders.remove(overviewProvider); + } + +//================================================================================================== +// Test methods +//================================================================================================== + BSimServerManager getServerManager() { + return serverManager; + } + + void setQueryServiceFactory(SFQueryServiceFactory factory) { + queryServiceFactory = factory; + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + abstract class AbstractProgramTask extends Task { + AbstractProgramTask(String title) { + super(title, true, true, false, false); + } + + abstract Program getProgram(); + } + + class OverviewTask extends AbstractProgramTask + implements SFResultsUpdateListener { + + private BSimSearchSettings settings; + private BSimServerCache serverCache; + private BSimOverviewProvider overviewProvider; + private SFOverviewInfo overviewInfo; + + public OverviewTask(BSimServerCache serverCache, BSimSearchSettings settings, + Set functions) { + super("Computing BSim Overview"); + + this.serverCache = serverCache; + this.settings = settings; + overviewInfo = new SFOverviewInfo(functions); + overviewInfo.setSimilarityThreshold(settings.getSimilarity()); + overviewInfo.setSignificanceThreshold(settings.getConfidence()); + } + + @Override + Program getProgram() { + return overviewInfo.getProgram(); + } + + @Override + public void run(TaskMonitor monitor) throws CancelledException { + + lastUsedServerCache = serverCache; + lastUsedSettings = settings; + + try (SimilarFunctionQueryService queryService = + getQueryService(overviewInfo.getProgram(), serverCache.getServerInfo())) { + monitor.setMessage("Calculating overview..."); + queryService.overviewSimilarFunctions(overviewInfo, this, monitor); + } + catch (QueryDatabaseException e) { + Msg.showError(this, null, "Error Performing BSim Overview", e.getMessage(), e); + } + } + + @Override + public void resultAdded(QueryResponseRecord partialResponse) { + SwingUtilities.invokeLater(() -> { + getProvider().overviewResultAdded((ResponseNearestVector) partialResponse); + }); + } + + @Override + public void setFinalResult(ResponseNearestVector result) { + if (result != null) { + SwingUtilities.invokeLater(() -> { + getProvider().setFinalOverviewResults(result); + }); + } + } + + private BSimOverviewProvider getProvider() { + if (overviewProvider == null) { + BSimServerInfo serverInfo = serverCache.getServerInfo(); + LSHVectorFactory lshVectorFactory = serverCache.getLSHVectorFactory(); + Program program = overviewInfo.getProgram(); + overviewProvider = + getOverviewProvider(serverInfo, program, lshVectorFactory, settings); + overviewProviders.add(overviewProvider); + } + return overviewProvider; + } + + } + + class SearchTask extends AbstractProgramTask implements SFResultsUpdateListener { + + private BSimServerCache serverCache; + private BSimSearchResultsProvider resultsProvider; + private SFQueryInfo queryInfo; + private BSimSearchSettings settings; + + public SearchTask(BSimServerCache serverCache, BSimSearchSettings settings, + Set functions) { + super("BSim Search For Similar Functions"); + this.serverCache = serverCache; + this.settings = settings; + queryInfo = new SFQueryInfo(functions); + queryInfo.setSimilarityThreshold(settings.getSimilarity()); + queryInfo.setSignificanceThreshold(settings.getConfidence()); + queryInfo.setMaximumResults(settings.getMaxResults()); + BSimFilter bsimFilter = queryInfo.getBsimFilter(); + BSimFilterSet bSimFilterSet = settings.getBSimFilterSet(); + bsimFilter.replaceWith(bSimFilterSet.getBSimFilter()); + + } + + @Override + Program getProgram() { + return queryInfo.getProgram(); + } + + @Override + public void run(TaskMonitor monitor) throws CancelledException { + + monitor.setMessage("Connecting to database..."); + try (SimilarFunctionQueryService queryService = + getQueryService(queryInfo.getProgram(), serverCache.getServerInfo())) { + monitor.setMessage("Querying database..."); + queryService.querySimilarFunctions(queryInfo, this, monitor); + } + catch (QueryDatabaseException e) { + Msg.showError(this, null, "Error Performing BSim Search", e.getMessage(), e); + } + } + + @Override + public void resultAdded(QueryResponseRecord partialResponse) { + // NOTE: QueryResultsProvider is unable to handle incremental results + } + + @Override + public void setFinalResult(SFQueryResult result) { + if (result != null) { + SwingUtilities.invokeLater(() -> { + getSearchProvider().setFinalQueryResults(result); + }); + } + } + + private BSimSearchResultsProvider getSearchProvider() { + if (resultsProvider == null) { + BSimServerInfo serverInfo = serverCache.getServerInfo(); + DatabaseInformation databaseInfo = serverCache.getDatabaseInformation(); + LSHVectorFactory lshVectorFactory = serverCache.getLSHVectorFactory(); + resultsProvider = getSearchResultsProvider(serverInfo, databaseInfo, + lshVectorFactory, queryInfo, settings); + searchResultsProviders.add(resultsProvider); + } + return resultsProvider; + } + + } + + class MyBSimSearchService implements BSimSearchService { + + @Override + public BSimServerInfo getLastUsedServer() { + return lastUsedServerCache == null ? null : lastUsedServerCache.getServerInfo(); + } + + @Override + public BSimSearchSettings getLastUsedSearchSettings() { + return lastUsedSettings; + } + + @Override + public void search(BSimServerCache serverCache, BSimSearchSettings settings, + Set functions) { + if (checkBusy()) { + return; + } + + lastUsedServerCache = serverCache; + lastUsedSettings = settings; + + SearchTask searchTask = new SearchTask(serverCache, settings, functions); + currentTask = searchTask; + searchTask.addTaskListener(taskListener); + TaskLauncher.launch(searchTask); + } + + @Override + public void performOverview(BSimServerCache serverCache, BSimSearchSettings settings) { + if (checkBusy()) { + return; + } + lastUsedServerCache = serverCache; + lastUsedSettings = settings; + + OverviewTask task = new OverviewTask(serverCache, settings, getOverviewFunctions()); + currentTask = task; + task.addTaskListener(taskListener); + TaskLauncher.launch(task); + } + + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ArchitectureBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ArchitectureBSimFilterType.java new file mode 100644 index 0000000000..ef4599d892 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ArchitectureBSimFilterType.java @@ -0,0 +1,86 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; +import java.util.List; +import java.util.stream.Collectors; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; +import ghidra.program.model.lang.LanguageDescription; +import ghidra.program.model.lang.LanguageService; +import ghidra.program.util.DefaultLanguageService; +import utility.function.Callback; + +/** + * A BsimFilterType for filtering functions based on a Ghidra computer architecture + * specification. + */ +public class ArchitectureBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "archequals"; + + public ArchitectureBSimFilterType() { + super("Architecture equals", XML_VALUE, "x86:LE:64:default"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + effect.setExeTable(); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.architecture=").append(resolution.id1); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addDocValue("String arch = doc['architecture'].value; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, "arch == params." + argName); + effect.addParam(argName, atom.value); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (value.equals(rec.getArchitecture())); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return new IDSQLResolution.Architecture(atom.value); + } + + @Override + public BSimValueEditor getEditor(List initialValues, Callback listener) { + List choices = getArchitectures(); + return new MultiChoiceBSimValueEditor(this, choices, initialValues, "Architecture", + listener); + } + + public static List getArchitectures() { + LanguageService service = DefaultLanguageService.getLanguageService(); + List languages = service.getLanguageDescriptions(true); + return languages.stream() + .map(l -> l.getLanguageID().toString()) + .collect(Collectors.toList()); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BSimFilterType.java new file mode 100755 index 0000000000..21394275bb --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BSimFilterType.java @@ -0,0 +1,391 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.io.IOException; +import java.io.Writer; +import java.sql.SQLException; +import java.util.*; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; +import ghidra.xml.XmlElement; +import utility.function.Callback; + +/** + * The base class for BSim filter types. Each filter type represents a different filter criteria + * that can be applied to a BSim Search query. They have a human readable description and a way + * to convert string values for the filter into SQL queries. + */ +public abstract class BSimFilterType implements Comparable { + private static List basis = null; + public static BSimFilterType BLANK = new BlankBSimFilterType(); + protected String label; // The description of this element in gui menus + protected String xmlval; // Tag name for serialization of filters + protected String hint; // The text that will show in the gui input field as a 'hint' + + /** + * + * @param label is the name used for display + * @param xmlval is the name used for XML serialization + * @param hint is the pop-up menu hint + */ + public BSimFilterType(String label, String xmlval, String hint) { + this.label = label; + this.xmlval = xmlval; + this.hint = hint; + } + + @Override + public String toString() { + return label; + } + + @Override + public int hashCode() { + return Objects.hash(label); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null) { + return false; + } + if (getClass() != obj.getClass()) { + return false; + } + BSimFilterType other = (BSimFilterType) obj; + return Objects.equals(hint, other.hint) && Objects.equals(label, other.label) && + Objects.equals(xmlval, other.xmlval); + } + + @Override + public int compareTo(BSimFilterType op2) { + return label.compareTo(op2.label); + } + + /** + * @return the tag name for serialization + */ + public String getXmlValue() { + return xmlval; + } + + /** + * @return the hint text + */ + public String getHint() { + return hint; + } + + /** + * @return true if this is a filter element based on callgraph information of functions + */ + public boolean isChildFilter() { + return false; + } + + /** + * @return true if this is a "blank" filter (i.e. an unused element within a gui) + */ + public boolean isBlank() { + return false; + } + + /** + * @return true if any id's relevant to this filter must be resolved relative to the local ColumnDatabase + */ + public boolean isLocal() { + return true; + } + + /** + * @return true if multiple filters of this type are allowed. + */ + public boolean isMultipleEntryAllowed() { + return true; + } + + /** + * @return true if multiple filters of this type should be OR'd. AND them otherwise. + */ + public boolean orMultipleEntries() { + return true; + } + + /** + * Save XML attributes corresponding to this template + * @param fwrite is the output stream + * @throws IOException for problems writing to the stream + */ + public void saveXml(Writer fwrite) throws IOException { + fwrite.append(" type=\"").append(xmlval).append('\"'); + } + + /** + * Construct a record describing the column id's that might need to be recovered before this filter + * element can be converted to an SQL clause + * @param atom is the specific FilterAtom to generate the record for + * @return the IDSQLResolution record or null if no ids need to be recovered + */ + public abstract IDSQLResolution generateIDSQLResolution(FilterAtom atom); + + /** + * Construct a record describing the document id's that might be needed before this filter + * element can be converted to an Elasticsearch filter script clause + * @param atom is the specific FilterAtom to generate the record for + * @return the record or null if no ids need to be recovered + */ + public IDElasticResolution generateIDElasticResolution(FilterAtom atom) { + return null; + } + + /** + * Gather all pieces to successfully convert this filter element into an SQL clause + * @param effect is SQLEffects container for this filter elements pieces and others + * @param atom holds the values for a particular instantiation of this filter element + * @param resolution is the IDResolution containing relevant row ids for the filter, which must have been precalculated + * @throws SQLException for errors building the SQL clause + */ + public abstract void gatherSQLEffect(SQLEffects effect, FilterAtom atom, + IDSQLResolution resolution) throws SQLException; + + /** + * Gather pieces necessary to emit this filter as part of an elasticsearch query document + * @param effect is the ElasticEffects container holding the pieces + * @param atom holds the values for a particular instantiation of this filter element + * @param resolution contains relevant ids for the filter, which must have been precalculated + * @throws ElasticException for errors building the JSON subdocument + */ + public abstract void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException; + + /** + * Given (multiple) clauses for a single filter type, combine into a single SQL where clause + * @param subClauses is the list of SQL clauses + * @return the combined clause + */ + public String buildSQLCombinedClause(List subClauses) { + String appender = orMultipleEntries() ? " OR " : " AND "; // OR or AND things together + StringBuilder orClause = new StringBuilder(); + orClause.append('('); + boolean printAppender = false; + for (String str : subClauses) { + if (printAppender) { + orClause.append(appender); + } + orClause.append(str); + printAppender = true; + } + orClause.append(')'); + return orClause.toString(); + } + + /** + * Given (multiple) clauses for a single filter type, combine into a single elasticsearch script conditional + * @param subClauses is the list of script clauses + * @return the combined clause + */ + public String buildElasticCombinedClause(List subClauses) { + String appender = orMultipleEntries() ? " || " : " && "; // OR or AND things together + StringBuilder clause = new StringBuilder(); + clause.append('('); + boolean printAppender = false; + for (String str : subClauses) { + if (printAppender) { + clause.append(appender); + } + clause.append(str); + printAppender = true; + } + clause.append(')'); + return clause.toString(); + } + + public BSimValueEditor getEditor(List initialValues, Callback listener) { + return new StringBSimValueEditor(this, initialValues, listener); + } + + /** + * Evaluate this filter for a specific ExecutableRecord and a specific filter -value- + * @param rec is the ExecutableRecord to filter against + * @param value is the String value for an instantiated filter + * @return true if this element would allow the ExecutableRecord to pass the filter + */ + public abstract boolean evaluate(ExecutableRecord rec, String value); + + /** + * Tests if the given string is a valid value for this filter type. + * @param value the value to test + * @return true if the given string is valid for this filter + */ + public boolean isValidValue(String value) { + return value != null && !value.isBlank(); + } + + /** + * Returns a normalized version of the given value for this filter. + * @param value the value to be normalized + * @return a normalized version of the given value for this filter + */ + public String normalizeValue(String value) { + return value.trim(); + } + + /** + * @return the Blank FilterTemplate + */ + public static BSimFilterType getBlank() { + buildFilterBasis(); + return basis.get(0); + } + + public static List getBaseFilters() { + buildFilterBasis(); + return basis; + } + + /** + * Convenience function for deserializing FilterTemplates + * @param el is the tag to deserialize + * @return the deserialized FilterTemplate + */ + public static BSimFilterType nameToType(XmlElement el) { + String attr = el.getAttribute("type"); + buildFilterBasis(); // Make sure generic filter list is built + for (BSimFilterType cur : basis) { + if (cur.xmlval.equals(attr)) { + return cur; + } + } + if (attr.equals(ExecutableNameBSimFilterType.XML_VALUE)) { + String subattr = el.getAttribute("subtype"); + return new ExecutableCategoryBSimFilterType(subattr); + } + else if (attr.equals(NotExecutableCategoryBSimFilterType.XML_VALUE)) { + String subattr = el.getAttribute("subtype"); + return new NotExecutableCategoryBSimFilterType(subattr); + } + else if (attr.equals(FunctionTagBSimFilterType.XML_VALUE)) { + String tagName = el.getAttribute("tagname"); + int flag = Integer.decode(el.getAttribute("flag")); + return new FunctionTagBSimFilterType(tagName, flag); + } + return basis.get(0); // Default template + } + + /** + * Build the static set of basic FilterTemplates + */ + private static void buildFilterBasis() { + if (basis != null) { + return; // Already built + } + basis = new ArrayList(); + basis.add(new BlankBSimFilterType()); + basis.add(new ExecutableNameBSimFilterType()); + basis.add(new NotExecutableNameBSimFilterType()); + basis.add(new Md5BSimFilterType()); + basis.add(new NotMd5BSimFilterType()); + basis.add(new ArchitectureBSimFilterType()); + basis.add(new NotArchitectureBSimFilterType()); + basis.add(new CompilerBSimFilterType()); + basis.add(new NotCompilerBSimFilterType()); + basis.add(new PathStartsBSimFilterType()); + basis.add(new HasNamedChildBSimFilterType()); + } + + /** + * Generate a possibly restricted/extended set of FilterTemplates + * @param info is database information which informs about which filters to create + * @param includeChildFilter toggles whether or not ChildFilters should be included in this particular set + * @return the list of filter templates + */ + public static List generateBsimFilters(DatabaseInformation info, + boolean includeChildFilter) { + List resFilters = new ArrayList(); + buildFilterBasis(); + + for (BSimFilterType template : basis) { + if (template.isChildFilter()) { + if (!includeChildFilter || (info != null && !info.trackcallgraph)) { + continue; + } + } + resFilters.add(template); + } + + // If the database explicitly names the date column for the database, + // this OVERRIDES the default "Ingest Date" column. + // There is only room for one date in the table row + if (info != null && info.dateColumnName != null && !info.dateColumnName.isEmpty()) { + DateEarlierBSimFilterType earlierTemplate = + new DateEarlierBSimFilterType(info.dateColumnName); // Create customized date filters + DateLaterBSimFilterType laterTemplate = + new DateLaterBSimFilterType(info.dateColumnName); + resFilters.add(earlierTemplate); // Add customized filters to list + resFilters.add(laterTemplate); + } + else { + // Otherwise, we add default date filters to list + DateEarlierBSimFilterType filt = new DateEarlierBSimFilterType("Ingest Date"); + resFilters.add(filt); + DateLaterBSimFilterType filt2 = new DateLaterBSimFilterType("Ingest Date"); + resFilters.add(filt2); + } + if (info != null && info.execats != null) { + for (String element : info.execats) { + ExecutableCategoryBSimFilterType filt = + new ExecutableCategoryBSimFilterType(element); + resFilters.add(filt); + NotExecutableCategoryBSimFilterType filt2 = + new NotExecutableCategoryBSimFilterType(element); + resFilters.add(filt2); + } + } + FunctionTagBSimFilterType filtFuncTag; + filtFuncTag = new FunctionTagBSimFilterType("KNOWN_LIBRARY", + FunctionTagBSimFilterType.KNOWN_LIBRARY_MASK); + resFilters.add(filtFuncTag); + filtFuncTag = new FunctionTagBSimFilterType("HAS_UNIMPLEMENTED", + FunctionTagBSimFilterType.HAS_UNIMPLEMENTED_MASK); + resFilters.add(filtFuncTag); + filtFuncTag = new FunctionTagBSimFilterType("HAS_BADDATA", + FunctionTagBSimFilterType.HAS_BADDATA_MASK); + resFilters.add(filtFuncTag); + if (info != null && info.functionTags != null) { + int flag = 1; + flag <<= FunctionTagBSimFilterType.RESERVED_BITS; // First bits are reserved + for (String element : info.functionTags) { + filtFuncTag = new FunctionTagBSimFilterType(element, flag); + resFilters.add(filtFuncTag); + flag <<= 1; + } + } + return resFilters; + } + + public String getLabel() { + return label; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BSimValueEditor.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BSimValueEditor.java new file mode 100644 index 0000000000..69d41d2c86 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BSimValueEditor.java @@ -0,0 +1,62 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.awt.Color; +import java.util.List; + +import javax.swing.JComponent; + +import generic.theme.GThemeDefaults.Colors; +import generic.theme.GThemeDefaults.Colors.Palette; + +/** + * Interface for BSim filter value editors. Some BSim editors can support multiple values, so the + * getValues, setValues methods all work on lists of strings. + */ +public interface BSimValueEditor { + public static final String FILTER_DELIMETER = ","; + public static final Color VALID_COLOR = Colors.BACKGROUND; + public static final Color INVALID_COLOR = Palette.getColor("mistyrose"); + + /** + * Sets the editor to the given string values. They are displayed in the GUI as comma separated + * values. + * @param values the values to be used as the current editor values + */ + public void setValues(List values); + + /** + * Returns the current set of editor values. + * @return the current set of editor values + */ + public List getValues(); + + /** + * returns the GUI component used to allow the user to see and change editor values. + * @return the GUI component used to allow the user to see and change editor values + */ + public JComponent getComponent(); + + /** + * Returns true if the editor has valid values as determined by the editor's corresponding + * {@link BSimFilterType#isValidValue}. + * @return true if the editor has valid values as determined by the editor's corresponding + * filter type. + */ + public boolean hasValidValues(); + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BlankBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BlankBSimFilterType.java new file mode 100644 index 0000000000..4fd21be699 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BlankBSimFilterType.java @@ -0,0 +1,75 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BSimFilterType that represents a non-specified filter. Used for the gui so that when adding + * a filter it doesn't have to default to some specific filter. + */ +public class BlankBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "blank"; + + @Override + public boolean isBlank() { + return true; + } + + public BlankBSimFilterType() { + super(" ", XML_VALUE, ""); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + // Blank does nothing + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + // Blank does nothing + + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return true; + } + + @Override + public boolean isValidValue(String value) { + return value == null || value.isBlank(); + } + + @Override + public String normalizeValue(String value) { + return null; + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BooleanBSimValueEditor.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BooleanBSimValueEditor.java new file mode 100644 index 0000000000..3b73e02943 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/BooleanBSimValueEditor.java @@ -0,0 +1,107 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.util.List; + +import javax.swing.*; + +import ghidra.util.Swing; +import utility.function.Callback; + +/** + * A BSimValueEditor for boolean filter values. + */ +public class BooleanBSimValueEditor implements BSimValueEditor { + + private Callback listener; + private JRadioButton trueButton; + private JRadioButton falseButton; + private BSimFilterType filterType; + private JPanel component; + + public BooleanBSimValueEditor(BSimFilterType filterType, List initialValues, + Callback listener) { + this.filterType = filterType; + this.listener = listener; + this.component = createInputPanel(); + setValues(initialValues); + } + + private void setValue(String value) { + value = filterType.normalizeValue(value); + if ("true".equals(value)) { + trueButton.setSelected(true); + } + else if ("false".equals(value)) { + falseButton.setSelected(true); + } + } + + @Override + public void setValues(List values) { + String value = "true"; + if (values != null && !values.isEmpty()) { + value = values.get(0); + } + setValue(value); + } + + @Override + public List getValues() { + if (trueButton.isSelected()) { + return List.of("true"); + } + return List.of("false"); + } + + @Override + public JComponent getComponent() { + return component; + } + + private void valueChanged() { + Swing.runLater(() -> listener.call()); + } + + private JPanel createInputPanel() { + JPanel panel = new JPanel(); + panel.setLayout(new BoxLayout(panel, BoxLayout.X_AXIS)); + panel.setBorder(BorderFactory.createEmptyBorder()); + trueButton = new JRadioButton("True"); + falseButton = new JRadioButton("False"); + + ButtonGroup group = new ButtonGroup(); + group.add(trueButton); + group.add(falseButton); + + // Fire off a change event whenever the user selects a radio button. + trueButton.addActionListener(e -> valueChanged()); + falseButton.addActionListener(e -> valueChanged()); + + panel.add(trueButton); + panel.add(falseButton); + + // Initialize the panel so the True radio button is selected. + setValue("true"); + return panel; + } + + @Override + public boolean hasValidValues() { + return true; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/CompilerBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/CompilerBSimFilterType.java new file mode 100644 index 0000000000..caee5cd992 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/CompilerBSimFilterType.java @@ -0,0 +1,112 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; +import java.util.*; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; +import ghidra.program.model.lang.*; +import ghidra.program.util.DefaultLanguageService; +import utility.function.Callback; + +/** + * A BsimFilterType for filtering functions based on a Ghidra compiler specification. + */ +public class CompilerBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "compequals"; + + public CompilerBSimFilterType() { + super("Compiler equals", XML_VALUE, "gcc"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + effect.setExeTable(); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.name_compiler=").append(resolution.id1); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addDocValue("String comp = doc['name_compiler'].value; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, "comp == params." + argName); + effect.addParam(argName, atom.value); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (value.equals(rec.getNameCompiler())); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return new IDSQLResolution.Compiler(atom.value); + } + + @Override + public BSimValueEditor getEditor(List initialValues, Callback listener) { + List choices = getCompilers(); + return new MultiChoiceBSimValueEditor(this, choices, initialValues, "Compiler", + listener); + } + + /** + * Returns a list of all known compilers. + * + * @return the list of compiler specs + */ + private static List getCompilers() { + List compilers = new ArrayList<>(); + + List languages = + DefaultLanguageService.getLanguageService().getLanguageDescriptions(true); + + Set compilerIds = new HashSet<>(); + for (LanguageDescription language : languages) { + LanguageCompilerSpecQuery query = + new LanguageCompilerSpecQuery(language.getProcessor(), language.getEndian(), + language.getSize(), language.getVariant(), null); + + for (LanguageCompilerSpecPair specPair : DefaultLanguageService.getLanguageService() + .getLanguageCompilerSpecPairs( + query)) { + try { + String compilerId = + specPair.getCompilerSpecDescription().getCompilerSpecID().getIdAsString(); + if (!compilerIds.contains(compilerId)) { + compilerIds.add(compilerId); + compilers.add(compilerId); + } + } + catch (LanguageNotFoundException | CompilerSpecNotFoundException e) { + // Just eat the exception - no need to rethrow or print an error + } + } + } + + return compilers; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateBSimFilterType.java new file mode 100644 index 0000000000..f17256e723 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateBSimFilterType.java @@ -0,0 +1,103 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.time.LocalDate; +import java.time.format.DateTimeFormatter; +import java.time.format.DateTimeParseException; +import java.util.Arrays; +import java.util.List; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * An abstract BsimFilterType for filtering on dates. + */ +public abstract class DateBSimFilterType extends BSimFilterType { + public static final List FORMATTERS = Arrays.asList( + DateTimeFormatter.ofPattern("yyyy MM dd"), DateTimeFormatter.ofPattern("yyyy-MM-dd"), + DateTimeFormatter.ofPattern("yyyy/MM/dd"), DateTimeFormatter.ofPattern("MMM dd, yyyy"), + DateTimeFormatter.ofPattern("MMMM dd, yyyy"), DateTimeFormatter.ofPattern("MM dd yyyy"), + DateTimeFormatter.ofPattern("MM-dd-yyyy"), DateTimeFormatter.ofPattern("MM/dd/yyyy"), + DateTimeFormatter.ofPattern("yyyy")); + + /** + * + * @param label is the display name of this date filter + * @param xmlval is the XML serialization name + * @param hint is the pop-up hint + */ + public DateBSimFilterType(String label, String xmlval, String hint) { + super(label, xmlval, hint); + } + + public DateBSimFilterType() { + super("default label", "", ""); + } + + @Override + public boolean isValidValue(String value) { + return formatDate(value) != null; + } + + @Override + public String normalizeValue(String value) { + LocalDate date = formatDate(value); + return date == null ? null : date.toString(); + } + + /** + * Uses the list of {@link DateTimeFormatter} instances created above to test + * the given date value. If a formatter can parse the text, a {@link LocalDate} + * object is returned. + * + * @param value is the date string to format + * @return the formatted LocalDate or null + */ + protected LocalDate formatDate(String value) { + for (DateTimeFormatter formatter : FORMATTERS) { + try { + return LocalDate.parse(value, formatter); + } + catch (DateTimeParseException e) { + // just go back and parse the next one + } + } + + return null; + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return false; + } + + /** + * @return false, since having more than one date filter is logically inconsistent. + */ + @Override + public boolean isMultipleEntryAllowed() { + return false; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateEarlierBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateEarlierBSimFilterType.java new file mode 100644 index 0000000000..a1f28801a0 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateEarlierBSimFilterType.java @@ -0,0 +1,91 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; +import java.text.SimpleDateFormat; +import java.time.LocalDate; +import java.time.ZoneId; +import java.util.Date; + +import ghidra.features.bsim.query.client.*; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering on functions in programs created before the filter date. + */ +public class DateEarlierBSimFilterType extends DateBSimFilterType { + public static final String XML_VALUE = "dateearlier"; + + public DateEarlierBSimFilterType(String sub) { + super(sub + " is earlier than", XML_VALUE, "1974-09-21 or 09/21/1974"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + LocalDate localDate = formatDate(atom.value); + if (localDate == null) { + return; + } + //localDate is a calendar date, i.e., does not specify hours/minutes/seconds + //the database records timestamps for executables, so we need to create a timestamp + //from localDate. The timestamp we create corresponds to midnight on localDate + //*in the system default time zone of the client*. If an executable's ingest timestamp is + //before this timestamp, it passes the filter. + Date date = + Date.from(localDate.atStartOfDay().atZone(ZoneId.systemDefault()).toInstant()); + effect.setExeTable(); + String dateString = + new SimpleDateFormat(AbstractSQLFunctionDatabase.JAVA_TIME_FORMAT).format(date); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.ingest_date < to_timestamp('") + .append(dateString) + .append("','" + AbstractSQLFunctionDatabase.SQL_TIME_FORMAT + "')"); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + LocalDate localDate = formatDate(atom.value); + Date date = + Date.from(localDate.atStartOfDay().atZone(ZoneId.systemDefault()).toInstant()); + if (date == null) { + return; + } + + effect.addDocValue("ZonedDateTime date = doc['ingest_date'].value; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, "ZonedDateTime.ofInstant(Instant.ofEpochMilli(params." + + argName + "), ZoneId.of('Z')).compareTo(date) > 0"); + effect.addDateParam(argName, date); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + LocalDate localDate = formatDate(value); + Date date = + Date.from(localDate.atStartOfDay().atZone(ZoneId.systemDefault()).toInstant()); + if (date == null) { + return true; // Don't filter anything if we can't get a date + } + return rec.getDate().before(date); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateLaterBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateLaterBSimFilterType.java new file mode 100644 index 0000000000..aef9f1ab13 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/DateLaterBSimFilterType.java @@ -0,0 +1,91 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; +import java.text.SimpleDateFormat; +import java.time.LocalDate; +import java.time.ZoneId; +import java.util.Date; + +import ghidra.features.bsim.query.client.*; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering on functions in programs created after the filter date. + */ +public class DateLaterBSimFilterType extends DateBSimFilterType { + public static final String XML_VALUE = "datelater"; + + public DateLaterBSimFilterType(String sub) { + super(sub + " is later than", XML_VALUE, "1974-09-21 or 09/21/1974"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + LocalDate localDate = formatDate(atom.value); + if (localDate == null) { + return; + } + //localDate is a calendar date, i.e., does not specify hours/minutes/seconds + //the database records timestamps for executables, so we need to create a timestamp + //from localDate. The timestamp we create corresponds to midnight on the day after + //localDate *in the system default time zone of the client*. If an executable's + //ingest timestamp is greater than or equal to this timestamp, it passes the filter + localDate = localDate.plusDays(1); + Date date = + Date.from(localDate.atStartOfDay().atZone(ZoneId.systemDefault()).toInstant()); + effect.setExeTable(); + String dateString = + new SimpleDateFormat(AbstractSQLFunctionDatabase.JAVA_TIME_FORMAT).format(date); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.ingest_date >= to_timestamp('") + .append(dateString) + .append("','" + AbstractSQLFunctionDatabase.SQL_TIME_FORMAT + "')"); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + LocalDate localDate = formatDate(atom.value); + Date date = + Date.from(localDate.atStartOfDay().atZone(ZoneId.systemDefault()).toInstant()); + if (date == null) { + return; + } + + effect.addDocValue("ZonedDateTime date = doc['ingest_date'].value; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, "ZonedDateTime.ofInstant(Instant.ofEpochMilli(params." + + argName + "), ZoneId.of('Z')).compareTo(date) <= 0"); + effect.addDateParam(argName, date); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + LocalDate localDate = formatDate(value); + Date date = + Date.from(localDate.atStartOfDay().atZone(ZoneId.systemDefault()).toInstant()); + if (date == null) { + return true; // Don't filter anything if we can't get a date + } + return rec.getDate().after(date); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ExecutableCategoryBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ExecutableCategoryBSimFilterType.java new file mode 100644 index 0000000000..3820d794d1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ExecutableCategoryBSimFilterType.java @@ -0,0 +1,129 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.io.IOException; +import java.io.Writer; +import java.sql.SQLException; +import java.util.List; +import java.util.Objects; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.facade.SimilarFunctionQueryService; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering functions based on specific category values. + */ +public class ExecutableCategoryBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "execatmatches"; + + private String subType; // The specific executable category this element filters on + + public ExecutableCategoryBSimFilterType(String sub) { + super(sub + " matches", XML_VALUE, "category value"); + subType = sub; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + super.saveXml(fwrite); + fwrite.append(" subtype=\"").append(subType).append('\"'); + } + + @Override + public int hashCode() { + final int prime = 31; + int result = super.hashCode(); + result = prime * result + Objects.hash(subType); + return result; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (!super.equals(obj)) { + return false; + } + if (obj instanceof ExecutableCategoryBSimFilterType t) { + return Objects.equals(subType, t.subType); + } + return false; + } + + /** + * Custom category filters are processed after results are received, as a necessary consequence + * of the database structure. So we allow the query to return all possible results, and cull + * them after the fact. + * + * @see SimilarFunctionQueryService + */ + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + StringBuilder buf = new StringBuilder(); + buf.append("(execattable.id_type=").append(resolution.id1); + buf.append(" AND execattable.id_category=").append(resolution.id2).append(')'); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addDocValue("def execat = doc['execategory']; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, + "Collections.binarySearch(execat,params." + argName + ")>=0"); + effect.addParam(argName, subType + "\\t" + atom.value); + } + + @Override + public String buildSQLCombinedClause(List subClauses) { + StringBuilder buf = new StringBuilder(); + buf.append( + "EXISTS ( SELECT 1 FROM execattable WHERE desctable.id_exe=execattable.id_exe AND ("); + boolean printOr = false; + for (String clause : subClauses) { + if (printOr) { + buf.append(" OR "); + } + buf.append(clause); + printOr = true; + } + buf.append(") )"); + return buf.toString(); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return rec.hasCategory(subType, value); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return new IDSQLResolution.ExeCategory(subType, atom.value); + } + + @Override + public boolean orMultipleEntries() { + return true; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ExecutableNameBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ExecutableNameBSimFilterType.java new file mode 100644 index 0000000000..e7f0934e00 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/ExecutableNameBSimFilterType.java @@ -0,0 +1,67 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; + +import org.json.simple.JSONObject; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering on functions by the name of their containing program. + */ +public class ExecutableNameBSimFilterType extends BSimFilterType { + + public static final String XML_VALUE = "nameequals"; + + public ExecutableNameBSimFilterType() { + super("Executable name equals", XML_VALUE, "executable name"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + effect.setExeTable(); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.name_exec = '").append(atom.value).append('\''); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("\"filter\": { \"term\": { \"name_exec\": \""); + buffer.append(JSONObject.escape(atom.value)); + buffer.append("\" } } "); + effect.addStandalone(this, buffer.toString()); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (value.equals(rec.getNameExec())); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; // No resolution needed + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/FunctionTagBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/FunctionTagBSimFilterType.java new file mode 100644 index 0000000000..2cbc8e5e34 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/FunctionTagBSimFilterType.java @@ -0,0 +1,178 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.io.IOException; +import java.io.Writer; +import java.sql.SQLException; +import java.util.List; +import java.util.Objects; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.facade.SimilarFunctionQueryService; +import ghidra.features.bsim.query.protocol.FilterAtom; +import ghidra.util.exception.InvalidInputException; +import utility.function.Callback; + +/** + * A BsimFilterType for filtering functions based on specific function tag values. + */ +public class FunctionTagBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "functiontag"; + public static int RESERVED_BITS = 3; + public static int MAX_TAG_COUNT = 32 - RESERVED_BITS; + public static int KNOWN_LIBRARY_MASK = 1; + public static int HAS_UNIMPLEMENTED_MASK = 2; + public static int HAS_BADDATA_MASK = 4; + private String tagName; // Particular tag being tested + private int flag; // bit position of the boolean value + + /** + * Creates a new function tag filter. + * + * @param tagName the tag name + * @param flag the bit position of this flag + */ + public FunctionTagBSimFilterType(String tagName, int flag) { + super("Function tagged as " + tagName, XML_VALUE, "function tag"); + this.tagName = tagName; + this.flag = flag; + } + + /** + * Constructor for clients who do not know what the bit flag position of this + * function tag is. If that's the case, this will figure it out from the + * given queryService object. + * + * @param tagName the name of the tag + * @param queryService query service used to retrieve tag big position + * @throws InvalidInputException thrown if tag does not exist + */ + public FunctionTagBSimFilterType(String tagName, SimilarFunctionQueryService queryService) + throws InvalidInputException { + super("Function tagged as " + tagName, XML_VALUE, "function tag"); + this.tagName = tagName; + + DatabaseInformation info = queryService.getDatabaseInformation(); + if (info == null) { + throw new IllegalStateException("queryService has not been initialized"); + } + List functionTags = info.functionTags; + if (functionTags == null) { + throw new InvalidInputException("Function tag does not exist: " + tagName); + } + + flag = 8; + for (String tag : functionTags) { + if (tag.endsWith(tagName)) { + break; + } + flag = flag * 2; + } + } + + public int getFlag() { + return flag; + } + + @Override + public int hashCode() { + final int prime = 31; + int result = super.hashCode(); + result = prime * result + Objects.hash(flag, tagName); + return result; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (!super.equals(obj)) { + return false; + } + if (obj instanceof FunctionTagBSimFilterType t) { + return flag == t.flag && Objects.equals(tagName, t.tagName); + } + return false; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + super.saveXml(fwrite); + fwrite.append(" tagname=\"").append(tagName).append('\"'); + fwrite.append(" flag=\"").append(Integer.toString(flag)).append('\"'); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return true; // Not a test on executable + } + + /** + * @return false, only one boolean value is allowed + */ + @Override + public boolean isMultipleEntryAllowed() { + return false; + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + effect.addFunctionFilter(flag, atom.value.equals("true")); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addFunctionFilter(flag, atom.value.equals("true")); + } + + @Override + public String normalizeValue(String value) { + if (value != null) { + if (value.equals("f") || value.equals("false")) { + return "false"; + } + if (value.equals("t") || value.equals("true")) { + return "true"; + } + } + return null; + } + + @Override + public boolean isValidValue(String value) { + return normalizeValue(value) != null; + } + + @Override + public BSimValueEditor getEditor(List initialValues, Callback listener) { + return new BooleanBSimValueEditor(this, initialValues, listener); + + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/HasNamedChildBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/HasNamedChildBSimFilterType.java new file mode 100644 index 0000000000..9498ed0766 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/HasNamedChildBSimFilterType.java @@ -0,0 +1,83 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.ChildAtom; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering functions based on calls to specific external functions. + * The called function must be external, i.e. in terms of the database, the function must be + * associated with a library executable (having no code body) + */ +public class HasNamedChildBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "namedchild"; + + public HasNamedChildBSimFilterType() { + super("Calls external function", XML_VALUE, "external subfunction"); + } + + @Override + public boolean isChildFilter() { + return true; + } + + @Override + public boolean isLocal() { + return false; + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + StringBuilder buf = new StringBuilder(); + buf.append("EXISTS (SELECT 1 FROM callgraphtable WHERE src = desctable.id AND dest = "); + buf.append(resolution.id1); + buf.append(')'); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + String argName = effect.assignArgument(); + effect.addChildId("Collections.binarySearch(childid,params." + argName + ")>=0"); + effect.addFuncParam(argName, resolution.idString); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return true; + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + ChildAtom childatom = (ChildAtom) atom; + return new IDSQLResolution.ExternalFunction(childatom.exename, childatom.name); + } + + @Override + public IDElasticResolution generateIDElasticResolution(FilterAtom atom) { + ChildAtom childatom = (ChildAtom) atom; + return new IDElasticResolution.ExternalFunction(childatom.exename, childatom.name); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/Md5BSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/Md5BSimFilterType.java new file mode 100644 index 0000000000..494558c878 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/Md5BSimFilterType.java @@ -0,0 +1,88 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering on functions by the md5 of their containing program. + */ +public class Md5BSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "md5equals"; + public static final String md5Regex = "[a-fA-F0-9]{32}"; + + public Md5BSimFilterType() { + super("MD5 equals", XML_VALUE, "32-digit hex value"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution res) + throws SQLException { + StringBuilder buf = new StringBuilder(); + effect.setExeTable(); + buf.append("exetable.md5 = '").append(atom.value).append('\''); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("\"filter\": { \"term\": { \"md5\": \""); + buffer.append(atom.value); + buffer.append("\" } } "); + effect.addStandalone(this, buffer.toString()); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (value.equals(rec.getMd5())); + } + + @Override + public String normalizeValue(String value) { + value = value.trim(); + if (value.length() == 34 && value.charAt(0) == '0' && value.charAt(1) == 'x') { + value = value.substring(2, 34); + } + if (value.length() != 32 || !value.matches(md5Regex)) { + return null; + } + value = value.toLowerCase(); + return value; + } + + @Override + public boolean isValidValue(String value) { + value = value.trim(); + if (value.length() == 34 && value.charAt(0) == '0' && value.charAt(1) == 'x') { + value = value.substring(2, 34); + } + return value.length() == 32 && value.matches(md5Regex); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; // Id resolution not needed + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/MultiChoiceBSimValueEditor.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/MultiChoiceBSimValueEditor.java new file mode 100644 index 0000000000..8071100c28 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/MultiChoiceBSimValueEditor.java @@ -0,0 +1,170 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.awt.BorderLayout; +import java.util.*; +import java.util.stream.Collectors; + +import javax.swing.*; +import javax.swing.event.DocumentEvent; +import javax.swing.event.DocumentListener; + +import docking.DockingWindowManager; +import docking.widgets.button.BrowseButton; +import docking.widgets.textfield.HintTextField; +import ghidra.util.Swing; +import ghidra.util.layout.MiddleLayout; +import utility.function.Callback; + +/** + * Base class for BSimValueEditors that work on a list of possible choices + */ +public class MultiChoiceBSimValueEditor implements BSimValueEditor { + private BSimFilterType filterType; + private Callback listener; + private HintTextField textField; + private JComponent component; + private List choices; + private String dataTitle; + private boolean isValid; + + public MultiChoiceBSimValueEditor(BSimFilterType filterType, List choices, + List initialValues, String dataTitle, Callback listener) { + + this.filterType = filterType; + this.choices = choices; + this.dataTitle = dataTitle; + this.listener = Callback.dummyIfNull(listener); + component = buildComponent(); + setValues(initialValues); + checkValid(); + } + + private JComponent buildComponent() { + JPanel panel = new JPanel(new BorderLayout()); + textField = createTextField(); + panel.add(textField, BorderLayout.CENTER); + panel.add(buildChooserButton(), BorderLayout.EAST); + return panel; + } + + private JPanel buildChooserButton() { + JPanel panel = new JPanel(new MiddleLayout()); + JButton button = new BrowseButton(); + button.addActionListener(e -> showChooser()); + panel.add(button); + return panel; + } + + private void showChooser() { + Set selected = new HashSet<>(getValues()); + MultiChoiceSelectionDialog dialog = + new MultiChoiceSelectionDialog<>(dataTitle, choices, selected); + DockingWindowManager.showDialog(dialog); + List selectedChoices = dialog.getSelectedChoices(); + if (selectedChoices != null) { + setValues(selectedChoices); + } + } + + private HintTextField createTextField() { + HintTextField hintField = new HintTextField(filterType.getHint()); + hintField.setColumns(20); + hintField.getDocument().addDocumentListener(new DocumentListener() { + + @Override + public void removeUpdate(DocumentEvent e) { + documentChanged(); + } + + @Override + public void insertUpdate(DocumentEvent e) { + documentChanged(); + } + + @Override + public void changedUpdate(DocumentEvent e) { + documentChanged(); + } + }); + return hintField; + } + + protected void documentChanged() { + Swing.runLater(() -> { + checkValid(); + listener.call(); + }); + } + + @Override + public void setValues(List values) { + if (values == null) { + textField.setText(""); + } + else { + String value = values.stream().collect(Collectors.joining(", ")); + textField.setText(value); + } + + } + + @Override + public List getValues() { + String text = textField.getText().trim(); + if (text.contains(",")) { + List values = new ArrayList<>(); + String[] vals = text.split(FILTER_DELIMETER); + for (String val : vals) { + if (val == null || val.isBlank()) { + continue; + } + values.add(val.trim()); + } + return values; + } + return List.of(text); + } + + @Override + public JComponent getComponent() { + return component; + } + + public boolean hasValidValues() { + return isValid; + } + + private void checkValid() { + isValid = checkForValidValues(); + textField.setBackground(isValid ? VALID_COLOR : INVALID_COLOR); + } + + private boolean checkForValidValues() { + List values = getValues(); + if (values == null || values.size() == 0) { + return filterType.isValidValue(""); + } + for (String string : values) { + if (!filterType.isValidValue(string)) { + return false; + } + } + return true; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/MultiChoiceSelectionDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/MultiChoiceSelectionDialog.java new file mode 100644 index 0000000000..a22827d40f --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/MultiChoiceSelectionDialog.java @@ -0,0 +1,201 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.awt.BorderLayout; +import java.util.*; + +import javax.swing.*; + +import docking.DialogComponentProvider; +import docking.widgets.DataToStringConverter; +import docking.widgets.table.*; +import ghidra.docking.settings.Settings; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.framework.plugintool.ServiceProviderStub; + +/** + * Dialog for selection one or more choices from a list of possible values. + * + * @param the type of choices + */ +public class MultiChoiceSelectionDialog extends DialogComponentProvider { + + private List selectedChoices; + private GFilterTable filterTable; + private ChoiceTableModel model; + + public MultiChoiceSelectionDialog(String dataTitle, List choices, Set selected) { + this(dataTitle, choices, selected, t -> t.toString()); + } + + public MultiChoiceSelectionDialog(String dataTitle, List choices, Set selected, + DataToStringConverter dataConverter) { + super(dataTitle + " Chooser"); + addWorkPanel(buildMainPanel(choices, selected, dataConverter, dataTitle)); + addOKButton(); + addCancelButton(); + } + + private JComponent buildMainPanel(List choices, Set selected, + DataToStringConverter dataConverter, String dataTitle) { + JPanel panel = new JPanel(new BorderLayout()); + panel.setBorder(BorderFactory.createEmptyBorder(10, 10, 10, 10)); + model = new ChoiceTableModel(choices, selected, dataConverter, dataTitle); + filterTable = new GFilterTable(model); + panel.add(filterTable); + return panel; + } + + @Override + protected void okCallback() { + selectedChoices = getSelectedChoicesFromTable(); + close(); + } + + private List getSelectedChoicesFromTable() { + return model.getSelectedData(); + } + + public List getSelectedChoices() { + return selectedChoices; + } + + class ChoiceRowObject { + private T data; + private boolean selected; + + ChoiceRowObject(T data, boolean selected) { + this.data = data; + this.selected = selected; + } + + public boolean isSelected() { + return selected; + } + + public T getData() { + return data; + } + + public void setSelected(boolean b) { + selected = b; + } + + } + + class ChoiceTableModel extends GDynamicColumnTableModel { + private List rows = new ArrayList<>(); + private DataToStringConverter stringConverter; + private String dataColumnTitle; + + ChoiceTableModel(List data, Set selected, DataToStringConverter stringConverter, + String dataColumnTitle) { + super(new ServiceProviderStub()); + this.stringConverter = stringConverter; + this.dataColumnTitle = dataColumnTitle; + for (T t : data) { + rows.add(new ChoiceRowObject(t, selected.contains(t))); + } + } + + @Override + public String getName() { + return "Chooser"; + } + + public List getSelectedData() { + List selected = new ArrayList<>(); + for (ChoiceRowObject row : rows) { + if (row.isSelected()) { + selected.add(row.getData()); + } + } + return selected; + } + + @Override + public List.ChoiceRowObject> getModelData() { + return rows; + } + + @Override + public boolean isCellEditable(int rowIndex, int columnIndex) { + return columnIndex == 0; + } + + @Override + public void setValueAt(Object aValue, int rowIndex, int columnIndex) { + boolean b = ((Boolean) aValue).booleanValue(); + rows.get(rowIndex).setSelected(b); + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = new TableColumnDescriptor<>(); + descriptor.addVisibleColumn(new SelectedColumn()); + descriptor.addVisibleColumn(new DataColumn(), 1, true); + return descriptor; + } + + @Override + public Object getDataSource() { + return null; + } + + private class SelectedColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Selected"; + } + + @Override + public Boolean getValue(ChoiceRowObject rowObject, Settings settings, Object data, + ServiceProvider provider) throws IllegalArgumentException { + + return rowObject.isSelected(); + } + + @Override + public int getColumnPreferredWidth() { + return 40; + } + } + + private class DataColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return dataColumnTitle; + } + + @Override + public String getValue(ChoiceRowObject rowObject, Settings settings, Object data, + ServiceProvider provider) throws IllegalArgumentException { + + return stringConverter.getString(rowObject.getData()); + } + + @Override + public int getColumnPreferredWidth() { + return 300; + } + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotArchitectureBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotArchitectureBSimFilterType.java new file mode 100644 index 0000000000..325b88ead7 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotArchitectureBSimFilterType.java @@ -0,0 +1,90 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; +import java.util.List; +import java.util.stream.Collectors; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; +import ghidra.program.model.lang.LanguageDescription; +import ghidra.program.model.lang.LanguageService; +import ghidra.program.util.DefaultLanguageService; +import utility.function.Callback; + +/** + * A BsimFilterType for filtering functions based on not matching a Ghidra computer architecture + * specification. + */ +public class NotArchitectureBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "archnotequal"; + + public NotArchitectureBSimFilterType() { + super("Architecture does not equal", XML_VALUE, "x86:LE:64:default"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + effect.setExeTable(); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.architecture != ").append(resolution.id1); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addDocValue("String arch = doc['architecture'].value; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, "arch != params." + argName); + effect.addParam(argName, atom.value); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (!value.equals(rec.getArchitecture())); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return new IDSQLResolution.Architecture(atom.value); + } + + @Override + public boolean orMultipleEntries() { + return false; + } + + @Override + public BSimValueEditor getEditor(List initialValues, Callback listener) { + List choices = getArchitectures(); + return new MultiChoiceBSimValueEditor(this, choices, initialValues, "Architecture", + listener); + } + + public static List getArchitectures() { + LanguageService service = DefaultLanguageService.getLanguageService(); + List languages = service.getLanguageDescriptions(true); + return languages.stream() + .map(l -> l.getLanguageID().toString()) + .collect(Collectors.toList()); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotCompilerBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotCompilerBSimFilterType.java new file mode 100644 index 0000000000..f2ef950f97 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotCompilerBSimFilterType.java @@ -0,0 +1,117 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; +import java.util.*; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; +import ghidra.program.model.lang.*; +import ghidra.program.util.DefaultLanguageService; +import utility.function.Callback; + +/** + * A BsimFilterType for filtering functions based on not matching a Ghidra compiler specification. + */ +public class NotCompilerBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "compnotequal"; + + public NotCompilerBSimFilterType() { + super("Compiler does not equal", XML_VALUE, "gcc"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + effect.setExeTable(); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.name_compiler!=").append(resolution.id1); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addDocValue("String comp = doc['name_compiler'].value; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, "comp != params." + argName); + effect.addParam(argName, atom.value); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (!value.equals(rec.getNameCompiler())); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return new IDSQLResolution.Compiler(atom.value); + } + + @Override + public boolean orMultipleEntries() { + return false; + } + + @Override + public BSimValueEditor getEditor(List initialValues, Callback listener) { + List choices = getCompilers(); + return new MultiChoiceBSimValueEditor(this, choices, initialValues, "Compiler", + listener); + } + + /** + * Returns a list of all known compilers. + * + * @return the list of compiler specs + */ + private static List getCompilers() { + List compilers = new ArrayList<>(); + + List languages = + DefaultLanguageService.getLanguageService().getLanguageDescriptions(true); + + Set compilerIds = new HashSet<>(); + for (LanguageDescription language : languages) { + LanguageCompilerSpecQuery query = + new LanguageCompilerSpecQuery(language.getProcessor(), language.getEndian(), + language.getSize(), language.getVariant(), null); + + for (LanguageCompilerSpecPair specPair : DefaultLanguageService.getLanguageService() + .getLanguageCompilerSpecPairs( + query)) { + try { + String compilerId = + specPair.getCompilerSpecDescription().getCompilerSpecID().getIdAsString(); + if (!compilerIds.contains(compilerId)) { + compilerIds.add(compilerId); + compilers.add(compilerId); + } + } + catch (LanguageNotFoundException | CompilerSpecNotFoundException e) { + // Just eat the exception - no need to rethrow or print an error + } + } + } + + return compilers; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotExecutableCategoryBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotExecutableCategoryBSimFilterType.java new file mode 100644 index 0000000000..793709e810 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotExecutableCategoryBSimFilterType.java @@ -0,0 +1,129 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.io.IOException; +import java.io.Writer; +import java.sql.SQLException; +import java.util.List; +import java.util.Objects; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.facade.SimilarFunctionQueryService; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering functions based on not matching specific category values. + */ +public class NotExecutableCategoryBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "execatnomatch"; + + private String subType; + + public NotExecutableCategoryBSimFilterType(String sub) { + super(sub + " does not match", XML_VALUE, "category value"); + subType = sub; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + super.saveXml(fwrite); + fwrite.append(" subtype=\"").append(subType).append('\"'); + } + + @Override + public int hashCode() { + final int prime = 31; + int result = super.hashCode(); + result = prime * result + Objects.hash(subType); + return result; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (!super.equals(obj)) { + return false; + } + if (obj instanceof NotExecutableCategoryBSimFilterType t) { + return Objects.equals(subType, t.subType); + } + return false; + } + + /** + * Custom category filters are processed after results are received, as a necessary consequence + * of the database structure. So we allow the query to return all possible results, and cull + * them after the fact. + * + * @see SimilarFunctionQueryService + */ + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + StringBuilder buf = new StringBuilder(); + buf.append("(execattable.id_type =").append(resolution.id1); + buf.append(" AND execattable.id_category =").append(resolution.id2).append(')'); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addDocValue("def execat = doc['execategory']; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, + "Collections.binarySearch(execat,params." + argName + ")<0"); + effect.addParam(argName, subType + "\\t" + atom.value); + } + + @Override + public String buildSQLCombinedClause(List subClauses) { + StringBuilder buf = new StringBuilder(); + buf.append( + "NOT EXISTS ( SELECT 1 FROM execattable WHERE desctable.id_exe=execattable.id_exe AND ("); + boolean printOr = false; + for (String clause : subClauses) { + if (printOr) { + buf.append(" OR "); + } + buf.append(clause); + printOr = true; + } + buf.append(") )"); + return buf.toString(); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return !rec.hasCategory(subType, value); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return new IDSQLResolution.ExeCategory(subType, atom.value); + } + + @Override + public boolean orMultipleEntries() { + return false; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotExecutableNameBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotExecutableNameBSimFilterType.java new file mode 100644 index 0000000000..ecca1462c0 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotExecutableNameBSimFilterType.java @@ -0,0 +1,71 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; + +import org.json.simple.JSONObject; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering on functions whose containing program don't match a specific name. + */ +public class NotExecutableNameBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "namenotequal"; + + public NotExecutableNameBSimFilterType() { + super("Executable name does not equal", XML_VALUE, "executable name"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + effect.setExeTable(); + StringBuilder buf = new StringBuilder(); + buf.append("exetable.name_exec != '").append(atom.value).append('\''); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("\"must_not\": { \"term\": { \"name_exec\": \""); + buffer.append(JSONObject.escape(atom.value)); + buffer.append("\" } } "); + effect.addStandalone(this, buffer.toString()); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (!value.equals(rec.getNameExec())); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; // Id resolution not needed + } + + @Override + public boolean orMultipleEntries() { + return false; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotMd5BSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotMd5BSimFilterType.java new file mode 100644 index 0000000000..cfe06fed2a --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/NotMd5BSimFilterType.java @@ -0,0 +1,90 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering on functions whose containing program don't match a specific md5. + */ +public class NotMd5BSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "md5notequal"; + + public NotMd5BSimFilterType() { + super("MD5 does not equal", XML_VALUE, "32-digit hex value"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + StringBuilder buf = new StringBuilder(); + effect.setExeTable(); + buf.append("exetable.md5 != '").append(atom.value).append('\''); + effect.addWhere(this, buf.toString()); + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("\"must_not\": { \"term\": { \"md5\": \""); + buffer.append(atom.value); + buffer.append("\" } } "); + effect.addStandalone(this, buffer.toString()); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + return (!value.equals(rec.getMd5())); + } + + @Override + public String normalizeValue(String value) { + if (value.length() == 34 && value.charAt(0) == '0' && value.charAt(1) == 'x') { + value = value.substring(2, 34); + } + if (value.length() != 32 || !value.matches(Md5BSimFilterType.md5Regex)) { + return null; + } + value = value.toLowerCase(); + return value; + } + + @Override + public boolean isValidValue(String value) { + value = value.trim(); + if (value.length() == 34 && value.charAt(0) == '0' && value.charAt(1) == 'x') { + value = value.substring(2, 34); + } + return value.length() == 32 && value.matches(Md5BSimFilterType.md5Regex); + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; // Id resolution not needed + } + + @Override + public boolean orMultipleEntries() { + return false; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/PathStartsBSimFilterType.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/PathStartsBSimFilterType.java new file mode 100644 index 0000000000..c047100669 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/PathStartsBSimFilterType.java @@ -0,0 +1,97 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.sql.SQLException; + +import ghidra.features.bsim.query.client.IDSQLResolution; +import ghidra.features.bsim.query.client.SQLEffects; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.elastic.*; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * A BsimFilterType for filtering on functions by the starting path of their containing program. + */ +public class PathStartsBSimFilterType extends BSimFilterType { + public static final String XML_VALUE = "pathstarts"; + + public PathStartsBSimFilterType() { + super("Path starts with", XML_VALUE, "path"); + } + + @Override + public void gatherSQLEffect(SQLEffects effect, FilterAtom atom, IDSQLResolution resolution) + throws SQLException { + if (atom.value.length() > 0) { + effect.setExeTable(); + effect.setPathTable(); + StringBuilder buf = new StringBuilder(); + buf.append("position( \'").append(atom.value).append("\' in pathtable.val) = 1"); + effect.addWhere(this, buf.toString()); + } + } + + @Override + public void gatherElasticEffect(ElasticEffects effect, FilterAtom atom, + IDElasticResolution resolution) throws ElasticException { + effect.addDocValue("String path = doc['path'].value; "); + String argName = effect.assignArgument(); + effect.addScriptElement(this, + "(path != null) && path.startsWith(params." + argName + ')'); + effect.addParam(argName, atom.value); + } + + @Override + public boolean evaluate(ExecutableRecord rec, String value) { + if (rec.getPath() == null) { + return false; + } + return rec.getPath().startsWith(value); + } + + @Override + public String normalizeValue(String value) { + value = value.trim(); + // If the string is empty, it's invalid, just return false. + if (value.isBlank()) { + return null; + } + int pos = 0; + int posend = value.length(); + if (value.charAt(0) == '/') { + pos = 1; + } + if (value.charAt(posend - 1) == '/') { + posend -= 1; + } + if (posend <= pos) { + return null; + } + value = value.substring(pos, posend); + return value; + } + + @Override + public boolean isValidValue(String value) { + return normalizeValue(value) != null; + } + + @Override + public IDSQLResolution generateIDSQLResolution(FilterAtom atom) { + return null; // Id resolution not needed + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/StringBSimValueEditor.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/StringBSimValueEditor.java new file mode 100644 index 0000000000..27e3db21f5 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/filters/StringBSimValueEditor.java @@ -0,0 +1,130 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.filters; + +import java.util.ArrayList; +import java.util.List; +import java.util.stream.Collectors; + +import javax.swing.JComponent; +import javax.swing.event.DocumentEvent; +import javax.swing.event.DocumentListener; + +import docking.widgets.textfield.HintTextField; +import ghidra.util.Swing; +import utility.function.Callback; + +/** + * A BSimValueEditor for filters with arbitrary string values. Supports comma separated values. + */ +public class StringBSimValueEditor implements BSimValueEditor { + + private BSimFilterType filterType; + private Callback listener; + private HintTextField textField; + private boolean isValid; + + public StringBSimValueEditor(BSimFilterType filterType, List initialValues, + Callback listener) { + this.filterType = filterType; + this.listener = Callback.dummyIfNull(listener); + textField = new HintTextField(filterType.getHint()); + textField.setColumns(20); + setValues(initialValues); + textField.getDocument().addDocumentListener(new DocumentListener() { + + @Override + public void removeUpdate(DocumentEvent e) { + documentChanged(); + } + + @Override + public void insertUpdate(DocumentEvent e) { + documentChanged(); + } + + @Override + public void changedUpdate(DocumentEvent e) { + documentChanged(); + } + }); + checkValid(); + } + + protected void documentChanged() { + Swing.runLater(() -> { + checkValid(); + listener.call(); + }); + } + + @Override + public void setValues(List values) { + if (values == null) { + textField.setText(""); + } + else { + String value = values.stream().collect(Collectors.joining(", ")); + textField.setText(value); + } + + } + + @Override + public List getValues() { + String text = textField.getText().trim(); + if (text.contains(",")) { + List values = new ArrayList<>(); + String[] vals = text.split(FILTER_DELIMETER); + for (String val : vals) { + if (val == null || val.isBlank()) { + continue; + } + values.add(val.trim()); + } + return values; + } + return List.of(text); + } + + @Override + public JComponent getComponent() { + return textField; + } + + public boolean hasValidValues() { + return isValid; + } + + private void checkValid() { + isValid = checkForValidValues(); + textField.setBackground(isValid ? VALID_COLOR : INVALID_COLOR); + } + + private boolean checkForValidValues() { + List values = getValues(); + if (values == null || values.size() == 0) { + return filterType.isValidValue(""); + } + for (String string : values) { + if (!filterType.isValidValue(string)) { + return false; + } + } + return true; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewModel.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewModel.java new file mode 100755 index 0000000000..d7d4bcd9a3 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewModel.java @@ -0,0 +1,253 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.overview; + +import java.awt.Component; +import java.util.ArrayList; +import java.util.List; + +import javax.swing.JLabel; +import javax.swing.JTable; +import javax.swing.table.TableModel; + +import docking.widgets.table.*; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.docking.settings.Settings; +import ghidra.features.bsim.gui.search.results.BSimMatchResultsModel; +import ghidra.features.bsim.query.protocol.ResponseNearestVector; +import ghidra.features.bsim.query.protocol.SimilarityVectorResult; +import ghidra.framework.plugintool.PluginTool; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Program; +import ghidra.util.datastruct.Accumulator; +import ghidra.util.exception.CancelledException; +import ghidra.util.table.AddressBasedTableModel; +import ghidra.util.table.column.AbstractGColumnRenderer; +import ghidra.util.table.column.GColumnRenderer; +import ghidra.util.table.field.AbstractProgramBasedDynamicTableColumn; +import ghidra.util.table.field.AddressTableColumn; +import ghidra.util.task.TaskMonitor; + +/** + * Table model for BSim Overview results + */ +public class BSimOverviewModel extends AddressBasedTableModel { + + static final int NAME_COL = 0; + static final int HIT_COL = 1; + static final int SELF_COL = 2; + static final int ADDRESS_COL = 3; + + private List results = new ArrayList(); + private LSHVectorFactory vectorFactory; + + BSimOverviewModel(PluginTool tool, Program program, LSHVectorFactory vFactory) { + super("Query Overview", tool, null, null); + vectorFactory = vFactory; + setProgram(program); + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = + new TableColumnDescriptor(); + + descriptor.addVisibleColumn( + DiscoverableTableUtils.adaptColumForModel(this, new AddressTableColumn()), 1, true); + descriptor.addVisibleColumn(new FuncNameColumn()); + descriptor.addVisibleColumn(new HitCountColumn()); + descriptor.addVisibleColumn(new SelfSignificanceColumn()); + descriptor.addHiddenColumn(new VectorHashColumn()); + return descriptor; + } + + @Override + public Address getAddress(int row) { + return getRowObject(row).getFunctionEntryPoint(); + } + + @Override + protected void doLoad(Accumulator accumulator, TaskMonitor monitor) + throws CancelledException { + if (results.isEmpty()) { + return; + } + + for (BSimOverviewRowObject row : results) { + accumulator.add(row); + } + } + + void addResult(ResponseNearestVector response) { + if (response == null) { + return; // not sure if this can happen + } + + for (SimilarityVectorResult result : response.result) { + Address addr = BSimMatchResultsModel.recoverAddress(result.getBase(), program); + BSimOverviewRowObject row = new BSimOverviewRowObject(result, addr, vectorFactory); + addObject(row); + } + } + + void reload(Program newProgram, ResponseNearestVector response) { + setProgram(newProgram); + if ((response == null) || response.result.isEmpty()) { + clear(); + return; + } + + results.clear(); + for (SimilarityVectorResult result : response.result) { + Address addr = BSimMatchResultsModel.recoverAddress(result.getBase(), program); + BSimOverviewRowObject row = new BSimOverviewRowObject(result, addr, vectorFactory); + results.add(row); + } + super.reload(); + } + + void clear() { + clearData(); + } + + //================================================================================================== + // Inner Classes + //================================================================================================== + + private static class FuncNameColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Function Name"; + } + + @Override + public String getValue(BSimOverviewRowObject rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getFunctionName(); + } + + @Override + public int getColumnPreferredWidth() { + return 400; + } + + } + + private static class HitCountColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Hit Count"; + } + + @Override + public Integer getValue(BSimOverviewRowObject rowObject, Settings settings, Program data, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getHitCount(); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + + } + + private static class SelfSignificanceColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Self Significance"; + } + + @Override + public Double getValue(BSimOverviewRowObject rowObject, Settings settings, Program data, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getSelfSignificance(); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + } + + private static class VectorHashColumn + extends AbstractProgramBasedDynamicTableColumn { + private LongHexRenderer hexRenderer = new LongHexRenderer(); + + @Override + public String getColumnName() { + return "Vector Hash"; + } + + @Override + public Long getValue(BSimOverviewRowObject rowObject, Settings settings, Program data, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getVectorHash(); + } + + @Override + public GColumnRenderer getColumnRenderer() { + return hexRenderer; + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + + } + + private static class LongHexRenderer extends AbstractGColumnRenderer { + + @Override + public Component getTableCellRendererComponent(GTableCellRenderingData data) { + + JLabel label = (JLabel) super.getTableCellRendererComponent(data); + label.setHorizontalAlignment(RIGHT); + Long value = (Long) data.getValue(); + + if (value != null) { + label.setText(getValueString(value)); + } + return label; + } + + @Override + protected void configureFont(JTable table, TableModel model, int column) { + setFont(fixedWidthFont); + } + + @Override + public String getFilterString(Long t, Settings settings) { + return getValueString(t); + } + + private String getValueString(Long v) { + if (v == null) { + return ""; + } + return String.format("%016X", v); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewProvider.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewProvider.java new file mode 100755 index 0000000000..5477d6a2d8 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewProvider.java @@ -0,0 +1,252 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.overview; + +import java.awt.BorderLayout; +import java.awt.Dimension; +import java.awt.event.MouseEvent; +import java.util.ArrayList; +import java.util.List; + +import javax.swing.JComponent; +import javax.swing.JPanel; + +import docking.ActionContext; +import docking.WindowPosition; +import docking.action.builder.ActionBuilder; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.context.ProgramActionContext; +import ghidra.app.events.ProgramSelectionPluginEvent; +import ghidra.app.services.GoToService; +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.search.dialog.BSimSearchSettings; +import ghidra.features.bsim.gui.search.results.BSimSearchInfoDisplayDialog; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.protocol.ResponseNearestVector; +import ghidra.framework.plugintool.ComponentProviderAdapter; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Program; +import ghidra.program.util.ProgramLocation; +import ghidra.program.util.ProgramSelection; +import ghidra.util.HelpLocation; +import ghidra.util.table.*; +import resources.Icons; + +/** + * ComponentProvider to display the results of a BSim Overview query + */ +public class BSimOverviewProvider extends ComponentProviderAdapter { + private static final String PROVIDER_WINDOW_GROUP = "bsim.overview"; + + private static final String NAME = "BSim Function Overview"; + + private JComponent component; + private BSimSearchPlugin plugin; + private Program program; + private BSimOverviewModel overviewModel; + private GhidraTable table; + + private BSimServerInfo serverInfo; + + private BSimSearchSettings settings; + + public BSimOverviewProvider(BSimSearchPlugin plugin, BSimServerInfo serverInfo, Program program, + LSHVectorFactory vFactory, BSimSearchSettings settings) { + super(plugin.getTool(), NAME, plugin.getName()); + this.plugin = plugin; + this.serverInfo = serverInfo; + this.program = program; + this.settings = settings; + + setHelpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "BSim_Overview_Results")); + setDefaultWindowPosition(WindowPosition.WINDOW); + setWindowGroup(PROVIDER_WINDOW_GROUP); + setWindowMenuGroup("BSim"); + + // do this before setTitle() so that the windowing updates properly + setTabText(program.getName() + " -- " + serverInfo); + setTitle(NAME); + setTransient(); + + component = buildComponent(vFactory); + + tool.addComponentProvider(this, true); + + createActions(); + updateSubTitle(); + } + + public Program getProgram() { + return program; + } + + @Override + public JComponent getComponent() { + return component; + } + + private void createActions() { + addLocalAction(new SelectionNavigationAction(plugin, table)); + HelpLocation help = + new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Overview_Search_Info_Action"); + new ActionBuilder("Search Info", getName()).toolBarIcon(Icons.INFO_ICON) + .helpLocation(help) + .onAction(c -> showSearchInfo()) + .buildAndInstallLocal(this); + + new ActionBuilder("Make Selection", getOwner()).popupMenuPath("Make Selection") + .description("Make a selection using selected rows") + .helpLocation( + new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Overview_Make_Selection")) + .toolBarIcon(Icons.MAKE_SELECTION_ICON) + .enabledWhen(c -> table.getSelectedRowCount() > 0) + .onAction(c -> makeSelection()) + .buildAndInstallLocal(this); + + new ActionBuilder("Overview BSim Search From Dialog", getOwner()) + .popupMenuPath("Search Selected Functions...") + .description( + "Displays the BSim Simliar Functions Search Dialog with the selected funtions.") + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, + "Overview_Initiate_Search_Dialog")) + .enabledWhen(c -> table.getSelectedRowCount() > 0) + .onAction(c -> initialBSimSearch(true)) + .buildAndInstallLocal(this); + + new ActionBuilder("Overview BSim Search", getOwner()) + .popupMenuPath("Search Selected Functions") + .description("Performs a BSim Similar Functions Search on the selected functions.") + .helpLocation( + new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Overview_Initiate_Search")) + .enabledWhen(c -> table.getSelectedRowCount() > 0) + .onAction(c -> initialBSimSearch(false)) + .buildAndInstallLocal(this); + } + + private void showSearchInfo() { + tool.showDialog(new BSimSearchInfoDisplayDialog(serverInfo, settings, true)); + } + + private void initialBSimSearch(boolean showDialog) { + List
selectedFunctionAddresses = new ArrayList<>(); + int[] selectedRows = table.getSelectedRows(); + List rowObjects = overviewModel.getRowObjects(selectedRows); + for (BSimOverviewRowObject rowObject : rowObjects) { + selectedFunctionAddresses.add(rowObject.getFunctionEntryPoint()); + } + plugin.doBSimSearch(program, selectedFunctionAddresses, showDialog); + } + + private void makeSelection() { + ProgramSelection selection = table.getProgramSelection(); + if (program == null || program.isClosed() || selection.getNumAddresses() == 0) { + return; + } + GoToService service = tool.getService(GoToService.class); + if (service != null) { + service.goTo(new ProgramLocation(program, selection.getMinAddress())); + } + tool.firePluginEvent(new ProgramSelectionPluginEvent(getName(), selection, program)); + } + + private void updateSubTitle() { + int rowCount = overviewModel.getRowCount(); + int unfilteredCount = overviewModel.getUnfilteredRowCount(); + + StringBuilder buf = new StringBuilder(); + buf.append(program.getName()); + buf.append(" ("); + buf.append(rowCount); + buf.append(" functions"); + if (rowCount != unfilteredCount) { + buf.append(" (out of ").append(unfilteredCount).append(')'); + } + buf.append(")"); + setSubTitle(buf.toString()); + } + + private JComponent buildComponent(LSHVectorFactory vectorFactory) { + JPanel panel = new JPanel(new BorderLayout()); + + overviewModel = new BSimOverviewModel(tool, program, vectorFactory); + GhidraFilterTable filterTable = + new GhidraFilterTable<>(overviewModel); + table = filterTable.getTable(); + table.getSelectionModel().addListSelectionListener(e -> notifyContextChanged()); + table.setNavigateOnSelectionEnabled(true); + + overviewModel.addTableModelListener(e -> updateSubTitle()); + + filterTable.installNavigation(tool); + + panel.setPreferredSize(new Dimension(600, 400)); + panel.add(filterTable, BorderLayout.CENTER); + return panel; + } + + private void notifyContextChanged() { + tool.contextChanged(this); + } + + public void overviewResultAdded(ResponseNearestVector result) { + overviewModel.addResult(result); + } + + public void setFinalOverviewResults(ResponseNearestVector result) { + overviewModel.reload(program, result); + + component.revalidate(); + } + + @Override + public void componentHidden() { + super.componentHidden(); + if (plugin != null) { + plugin.providerClosed(this); + } + } + + @Override + public ActionContext getActionContext(MouseEvent event) { + if (program == null) { + return null; + } + return new ProgramActionContext(this, program, table); + } + + @Override + public String toString() { + return getTitle(); + } + + @Override + public final int hashCode() { + return super.hashCode(); + } + + @Override + public final boolean equals(Object obj) { + return super.equals(obj); + } + +//================================================================================================== +// Test methods +//================================================================================================== + BSimOverviewModel getModel() { + return overviewModel; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewRowObject.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewRowObject.java new file mode 100755 index 0000000000..231c8dfe6b --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewRowObject.java @@ -0,0 +1,70 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.overview; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.features.bsim.query.description.SignatureRecord; +import ghidra.features.bsim.query.protocol.SimilarityVectorResult; +import ghidra.program.model.address.Address; + +/** + * Table row object for BSim Overview results table + */ +public class BSimOverviewRowObject { + private Address addr; + private FunctionDescription func; + private SimilarityVectorResult simvec; + private double selfsignif; // Maximum significance score a query with this function could return + private long vectorHash; + + public BSimOverviewRowObject(SimilarityVectorResult result,Address ad,LSHVectorFactory vectorFactory) { + addr = ad; + simvec = result; + func = simvec.getBase(); + selfsignif = 0.0; + SignatureRecord sigrec = func.getSignatureRecord(); + if (sigrec != null) { + selfsignif = vectorFactory.getSelfSignificance(sigrec.getLSHVector()); + } + LSHVector lshVector = func.getSignatureRecord().getLSHVector(); + vectorHash = lshVector.calcUniqueHash(); + + } + + public String getFunctionName() { + return func.getFunctionName(); + } + + public Address getFunctionEntryPoint() { + return addr; + } + + public int getHitCount() { + return simvec.getTotalCount(); + } + + public double getSelfSignificance() { + return selfsignif; + } + + public long getVectorHash() { + return vectorHash; + } + + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewRowObjectToAddressTableRowMapper.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewRowObjectToAddressTableRowMapper.java new file mode 100644 index 0000000000..e1ff36a8e3 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/overview/BSimOverviewRowObjectToAddressTableRowMapper.java @@ -0,0 +1,35 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.overview; + +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Program; +import ghidra.util.table.ProgramLocationTableRowMapper; + +/** + * Row object mapper for mapping BSimOverviewRowObjects to Addresses + */ +public class BSimOverviewRowObjectToAddressTableRowMapper + extends ProgramLocationTableRowMapper { + + @Override + public Address map(BSimOverviewRowObject rowObject, Program data, + ServiceProvider serviceProvider) { + return rowObject.getFunctionEntryPoint(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/AbstractBSimSearchDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/AbstractBSimSearchDialog.java new file mode 100644 index 0000000000..22ea7eb78e --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/AbstractBSimSearchDialog.java @@ -0,0 +1,291 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.*; +import java.awt.event.ItemEvent; +import java.awt.event.ItemListener; +import java.util.*; +import java.util.List; + +import javax.swing.*; + +import docking.DialogComponentProvider; +import docking.DockingWindowManager; +import docking.widgets.EmptyBorderButton; +import docking.widgets.combobox.GComboBox; +import docking.widgets.textfield.FloatingPointTextField; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.facade.QueryDatabaseException; +import ghidra.framework.plugintool.PluginTool; +import ghidra.util.Msg; +import ghidra.util.Swing; +import ghidra.util.layout.PairLayout; +import ghidra.util.layout.VerticalLayout; +import ghidra.util.task.Task; +import resources.Icons; + +/** + * Base class for BSim Search dialogs that all have a server comboBox, and entries for the + * similarity and confidence values. + */ +public abstract class AbstractBSimSearchDialog extends DialogComponentProvider { + private final BSimServerManager serverManager; + private GComboBox serverCombo; + private BSimServerManagerListener serverInfoListener = this::serverListChanged; + private ItemListener serverComboListener = this::comboChanged; + + protected BSimSearchService searchService; + protected PluginTool tool; + protected FloatingPointTextField similarityField; + protected FloatingPointTextField confidenceField; + protected BSimServerCache serverCache; // non-null when valid server is connected + + protected AbstractBSimSearchDialog(String title, PluginTool tool, BSimSearchService service, + BSimServerManager serverManager) { + + super(title, true); + this.tool = tool; + this.searchService = service; + this.serverManager = serverManager; + serverManager.addListener(serverInfoListener); + addWorkPanel(buildMainPanel()); + populateComboServerComboBox(); + + addOKButton(); + addCancelButton(); + + initializeConnection(searchService.getLastUsedServer()); + initializeSettings(searchService.getLastUsedSearchSettings()); + } + + protected void initializeSettings(BSimSearchSettings lastUsedSearchSettings) { + similarityField.setValue(lastUsedSearchSettings.getSimilarity()); + confidenceField.setValue(lastUsedSearchSettings.getConfidence()); + } + + private JPanel buildMainPanel() { + JPanel panel = new JPanel(new BorderLayout()); + panel.setBorder(BorderFactory.createEmptyBorder(10, 10, 10, 10)); + panel.add(buildNorthPanel(), BorderLayout.NORTH); + panel.add(buildCenterPanel(), BorderLayout.CENTER); + return panel; + } + + private Component buildNorthPanel() { + JPanel panel = new JPanel(new VerticalLayout(10)); + panel.add(buildServerPanel()); + panel.add(createTitledPanel("Options", buildOptionsPanel(), false)); + return panel; + } + + protected Component buildCenterPanel() { + return new JPanel(); + } + + protected JPanel buildServerPanel() { + JPanel panel = new JPanel(new PairLayout(10, 10)); + panel.add(new JLabel("BSim Server:")); + panel.add(buildServerComponent()); + return panel; + } + + protected JPanel buildOptionsPanel() { + + JPanel panel = new JPanel(new PairLayout(2, 10)); + + similarityField = new FloatingPointTextField(10); + similarityField.setValue(0.7); + similarityField.setMinValue(0.0); + similarityField.setMaxValue(1.0); + + confidenceField = new FloatingPointTextField(10); + confidenceField.setValue(0); + confidenceField.setMinValue(0.0); + + panel.add(new JLabel("Similarity Threshold (0-1):")); + panel.add(similarityField); + panel.add(new JLabel("Confidence Threshold:")); + panel.add(confidenceField); + return panel; + } + + @Override + public void dispose() { + serverManager.removeListener(serverInfoListener); + super.dispose(); + } + + private void serverListChanged() { + populateComboServerComboBox(); + } + + private void populateComboServerComboBox() { + serverCombo.removeItemListener(serverComboListener); + + DefaultComboBoxModel model = new DefaultComboBoxModel<>(); + + List serverInfos = new ArrayList<>(serverManager.getServerInfos()); + Collections.sort(serverInfos); + model.addElement(null); // allow no-selection (null entry / disconnected state) + model.addAll(serverInfos); + serverCombo.setModel(model); + serverCombo.addItemListener(serverComboListener); + } + + private void comboChanged(ItemEvent e) { + if (e.getStateChange() != ItemEvent.SELECTED) { + return; + } + Swing.runLater(() -> { + BSimServerInfo selected = (BSimServerInfo) serverCombo.getSelectedItem(); + if (serverCache == null || !serverCache.getServerInfo().equals(selected)) { + initializeConnection(selected); + } + }); + setStatusText(""); + } + + private void initializeConnection(BSimServerInfo info) { + if (info != null) { + try { + setServerCache(new BSimServerCache(info)); + return; + } + catch (QueryDatabaseException e) { + if (!e.getMessage().contains("cancelled")) { + Msg.showError(this, rootPanel, "BSim Server Connection Failure", + e.getMessage()); + } + } + catch (Exception e) { + Msg.showError(this, rootPanel, "BSim Server Connection Failure", + "Unexpected error while connecting to: " + info, e); + } + } + setServerCache(null); + } + + protected DatabaseInformation getDatabaseInformation() { + if (serverCache != null) { + return serverCache.getDatabaseInformation(); + } + return null; + } + + protected void updateSearchEnablement() { + setOkEnabled(canQuery()); + } + + protected boolean canQuery() { + if (serverCache == null) { + setStatusText("Please select a Bsim Server."); + return false; + } + return true; + } + + protected void setServerCache(BSimServerCache serverCache) { + this.serverCache = serverCache; + setSelectedComboValue(serverCache != null ? serverCache.getServerInfo() : null); + updateSearchEnablement(); + } + + private void setSelectedComboValue(BSimServerInfo info) { + if (!Objects.equals(serverCombo.getSelectedItem(), info)) { + serverCombo.removeItemListener(serverComboListener); + serverCombo.setSelectedItem(info); + serverCombo.addItemListener(serverComboListener); + } + } + + private JPanel buildServerComponent() { + JPanel panel = new JPanel(new BorderLayout()); + JPanel comboPanel = new JPanel(new BorderLayout()); + comboPanel.setBorder(BorderFactory.createEmptyBorder(0, 0, 0, 10)); + serverCombo = new GComboBox<>(); + serverCombo.addItemListener(serverComboListener); + comboPanel.add(serverCombo, BorderLayout.CENTER); + panel.add(comboPanel, BorderLayout.CENTER); + + JButton button = new EmptyBorderButton(Icons.CONFIGURE_FILTER_ICON); + button.setToolTipText("Show Server Manager Dialog"); + button.addActionListener(e -> managerServers()); + panel.add(button, BorderLayout.EAST); + return panel; + } + + protected JPanel createTitledPanel(String name, JComponent comp, boolean fullWidth) { + JPanel panel = new JPanel(new BorderLayout()); + panel.setBorder(BorderFactory.createEmptyBorder(20, 5, 0, 0)); + JPanel titlePanel = new JPanel(new BorderLayout()); + JPanel contentPanel = new JPanel(new BorderLayout()); + + panel.add(titlePanel, BorderLayout.NORTH); + panel.add(contentPanel, BorderLayout.CENTER); + + contentPanel.setBorder(BorderFactory.createEmptyBorder(10, 15, 0, 0)); + + contentPanel.add(comp, fullWidth ? BorderLayout.CENTER : BorderLayout.WEST); + + JLabel label = new JLabel(name); + label.setFont(label.getFont().deriveFont(Font.BOLD)); + titlePanel.add(label, BorderLayout.NORTH); + + return panel; + } + + private void managerServers() { + BSimServerDialog dialog = new BSimServerDialog(tool, serverManager); + DockingWindowManager.showDialog(dialog); + BSimServerInfo lastAddedServer = dialog.getLastAdded(); + if (lastAddedServer != null) { + initializeConnection(lastAddedServer); + } + } + +//================================================================================================== +// test methods +//================================================================================================== + protected void setServer(BSimServerInfo info) { + initializeConnection(info); + } + + protected BSimServerInfo getServer() { + return serverCache == null ? null : serverCache.getServerInfo(); + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + protected abstract class BSimQueryTask extends Task { + protected Exception errorException; + + BSimQueryTask(String title) { + super(title, true, true, false); + } + + boolean hasError() { + return errorException != null; + } + + Exception getError() { + return errorException; + } + + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimFilterPanel.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimFilterPanel.java new file mode 100644 index 0000000000..412596ba02 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimFilterPanel.java @@ -0,0 +1,210 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.*; +import java.util.ArrayList; +import java.util.List; +import java.util.function.Consumer; + +import javax.swing.*; + +import docking.widgets.EmptyBorderButton; +import generic.theme.GThemeDefaults.Colors.Viewport; +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.gui.search.dialog.BSimFilterSet.FilterEntry; +import ghidra.util.layout.VerticalLayout; +import resources.Icons; +import utility.function.Callback; + +/** + * Panel for specifying and managing BSim filters. + */ +public class BSimFilterPanel extends JPanel { + + private JPanel mainPanel; + private List filters; + private List filterWidgets = new ArrayList<>(); + private Consumer removeMeConsumer = this::removeFilterWidget; + private Callback changeListener; + + /** + * Constructs a filer panel with no filters + * @param changeListener the callback when filters change + */ + public BSimFilterPanel(Callback changeListener) { + this(List.of(BSimFilterType.BLANK), new BSimFilterSet(), changeListener); + } + + /** + * Constructs a filer panel with existing filters + * @param filters the list of filterTypes to display in the comboBox + * @param filterSet the current filter settings + * @param changeListener the callback when filters change + */ + public BSimFilterPanel(List filters, BSimFilterSet filterSet, + Callback changeListener) { + super(new BorderLayout()); + this.changeListener = changeListener; + mainPanel = new ScrollablePanel(); + JScrollPane scroll = new JScrollPane(mainPanel); + scroll.getViewport().setBackground(Viewport.UNEDITABLE_BACKGROUND); + add(scroll, BorderLayout.CENTER); + add(buildButtonPanel(), BorderLayout.EAST); + this.filters = filters; + setFilterSet(filterSet); + } + + /** + * Sets the panel to have the given filters + * @param filterSet the set of filters to show in the panel + */ + public void setFilterSet(BSimFilterSet filterSet) { + filterWidgets.removeAll(filterWidgets); + mainPanel.removeAll(); + List filterEntries = filterSet.getFilterEntries(); + + for (FilterEntry filterEntry : filterEntries) { + FilterWidget widget = new FilterWidget(filters, removeMeConsumer, changeListener); + widget.setFilter(filterEntry.filterType(), filterEntry.values()); + addFilterWidget(widget); + } + if (filterWidgets.isEmpty()) { + addFilterWidget(); + } + } + + /** + * Sets the choices for filter types in the filter comboBoxes. + * @param filters the filter types the user can choose + */ + public void setFilters(List filters) { + if (filters == null || filters.isEmpty()) { + filters = List.of(BSimFilterType.BLANK); + } + this.filters = filters; + for (FilterWidget widget : filterWidgets) { + widget.setFilters(this.filters); + } + } + + /** + * Returns the set of valid filters that are displayed in this filter panel + * @return the set of valid filters that are displayed in this filter panel + */ + public BSimFilterSet getFilterSet() { + BSimFilterSet set = new BSimFilterSet(); + for (FilterWidget filter : filterWidgets) { + if (!filter.isBlank() && filter.hasValidValue()) { + set.addEntry(filter.getSelectedFilter(), filter.getValues()); + } + } + return set; + } + + /** + * Returns true the panel has only valid filters. (Blank filter is ok) + * @return true the panel has only valid filters + */ + public boolean hasValidFilters() { + for (FilterWidget filter : filterWidgets) { + if (!filter.isBlank() && !filter.hasValidValue()) { + return false; + } + } + return true; + } + + private Component buildButtonPanel() { + JPanel panel = new JPanel(new VerticalLayout(10)); + panel.setBorder(BorderFactory.createEmptyBorder(10, 10, 0, 0)); + JButton addFilterButton = new EmptyBorderButton(Icons.ADD_ICON); + addFilterButton.setToolTipText("Add filter"); + addFilterButton.setName("Add Filter"); + addFilterButton.addActionListener(e -> addFilterWidget()); + panel.add(addFilterButton); + return panel; + } + + private void addFilterWidget() { + addFilterWidget(new FilterWidget(filters, removeMeConsumer, changeListener)); + validate(); + } + + private void addFilterWidget(FilterWidget widget) { + filterWidgets.add(widget); + mainPanel.add(widget); + } + + private void removeFilterWidget(FilterWidget widget) { + filterWidgets.remove(widget); + mainPanel.remove(widget); + if (mainPanel.getComponentCount() == 0) { + addFilterWidget(); + } + changeListener.call(); + validate(); + + } + +//================================================================================================== +// Test methods +//================================================================================================== + List getFilterWidgets() { + return filterWidgets; + } + +//================================================================================================== +// Inner classes +//================================================================================================== + private class ScrollablePanel extends JPanel implements Scrollable { + public ScrollablePanel() { + super(new VerticalLayout(5)); + setBorder(BorderFactory.createEmptyBorder(10, 10, 10, 10)); + } + + @Override + public Dimension getPreferredScrollableViewportSize() { + Dimension preferredSize = getPreferredSize(); + preferredSize.height = 100; + return preferredSize; + } + + @Override + public int getScrollableUnitIncrement(Rectangle visibleRect, int orientation, + int direction) { + return 10; + } + + @Override + public int getScrollableBlockIncrement(Rectangle visibleRect, int orientation, + int direction) { + return 20; + } + + @Override + public boolean getScrollableTracksViewportWidth() { + return true; + } + + @Override + public boolean getScrollableTracksViewportHeight() { + return false; + } + + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimFilterSet.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimFilterSet.java new file mode 100644 index 0000000000..8ac95a7e55 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimFilterSet.java @@ -0,0 +1,103 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.util.*; + +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.query.protocol.BSimFilter; + +/** + * Maintains the set of current filters in a nicer way than BSimFiler which breaks them down into + * filter pieces that doesn't maintain any order. + */ +public class BSimFilterSet { + List filterEntries = new ArrayList<>(); + + public BSimFilterSet() { + + } + + private BSimFilterSet(BSimFilterSet set) { + filterEntries.addAll(set.filterEntries); + } + + /** + * Adds a filter entry to this set of filters + * @param filterType the BSimFilterType for the added filter + * @param values the list of values for the given filter type + */ + public void addEntry(BSimFilterType filterType, List values) { + filterEntries.add(new FilterEntry(filterType, values)); + } + + /** + * Returns the number of filter entries in this filter set. + * @return the number of filter entries in this filter set + */ + public int size() { + return filterEntries.size(); + } + + /** + * Returns the corresponding BSimFilter for this FilterSet. + * @return the corresponding BSimFilter for this FilterSet + */ + public BSimFilter getBSimFilter() { + BSimFilter bsimFilter = new BSimFilter(); + for (FilterEntry filterEntry : filterEntries) { + BSimFilterType filterType = filterEntry.filterType; + List values = filterEntry.values; + + for (String filterVal : values) { + bsimFilter.addAtom(filterType, filterVal.trim()); + } + } + return bsimFilter; + } + + /** + * Returns a copy of this FilterSet. + * @return a copy of this FilterSet + */ + public BSimFilterSet copy() { + return new BSimFilterSet(this); + } + + /** + * Returns the filter entries contains in this FilterSet. + * @return the filter entries contains in this FilterSet + */ + public List getFilterEntries() { + return filterEntries; + } + + /** + * Removes all filter entries for the given FilterType. + * @param filterType the type of filters to be removed from this set + */ + public void removeAll(BSimFilterType filterType) { + Iterator it = filterEntries.iterator(); + while (it.hasNext()) { + if (it.next().filterType.equals(filterType)) { + it.remove(); + } + } + } + + // A record that represents a single filter value (filter type and its values) + public record FilterEntry(BSimFilterType filterType, List values) {/**/} +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimOverviewDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimOverviewDialog.java new file mode 100644 index 0000000000..2b44b19bbb --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimOverviewDialog.java @@ -0,0 +1,55 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.framework.plugintool.PluginTool; +import ghidra.program.model.listing.Program; +import ghidra.util.HelpLocation; + +/** + * Dialog for initiating a BSim overview query. + */ +public class BSimOverviewDialog extends AbstractBSimSearchDialog { + private Program program; + + public BSimOverviewDialog(PluginTool tool, BSimSearchService service, + BSimServerManager serverManager) { + super("BSim Overview", tool, service, serverManager); + setOkButtonText("Overview"); + setHelpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "BSim_Overview_Dialog")); + } + + @Override + protected void okCallback() { + searchService.performOverview(serverCache, getSearchSettings()); + close(); + } + + protected BSimSearchSettings getSearchSettings() { + BSimSearchSettings lastUsed = searchService.getLastUsedSearchSettings(); + double similarity = similarityField.getValue(); + double confidence = confidenceField.getValue(); + int maxResults = lastUsed.getMaxResults(); + BSimFilterSet filterSet = lastUsed.getBSimFilterSet().copy(); + return new BSimSearchSettings(similarity, confidence, maxResults, filterSet); + } + + public Program getProgram() { + return program; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchDialog.java new file mode 100644 index 0000000000..e7caa8469d --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchDialog.java @@ -0,0 +1,191 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.BorderLayout; +import java.awt.Component; +import java.math.BigInteger; +import java.util.List; +import java.util.Set; + +import javax.swing.*; + +import docking.DialogComponentProvider; +import docking.DockingWindowManager; +import docking.widgets.EmptyBorderButton; +import docking.widgets.textfield.IntegerTextField; +import generic.theme.GIcon; +import ghidra.app.services.GoToService; +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.framework.plugintool.PluginTool; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.util.HelpLocation; +import ghidra.util.MessageType; + +/** + * Dialog for initiating a BSim similar function match search. + */ +public class BSimSearchDialog extends AbstractBSimSearchDialog { + private static final Icon FUNCTIONS_ICON = new GIcon("icon.bsim.functions.table"); + + protected Set selectedFunctions; + private JTextField functionsField; + private BSimFilterPanel filterPanel; + + // Query Settings + private IntegerTextField maxResultsField; + + public BSimSearchDialog(PluginTool tool, BSimSearchService service, + BSimServerManager serverManager, Set functions) { + super("Bsim Search Dialog", tool, service, serverManager); + selectedFunctions = functions; + setHelpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "BSim_Search_Dialog")); + setMinimumSize(500, 400); + updateSearchFunctionsLabel(); + setOkButtonText("Search"); + } + + @Override + protected void initializeSettings(BSimSearchSettings lastUsedSearchSettings) { + super.initializeSettings(lastUsedSearchSettings); + maxResultsField.setValue(lastUsedSearchSettings.getMaxResults()); + filterPanel.setFilterSet(lastUsedSearchSettings.getBSimFilterSet()); + } + + @Override + protected void okCallback() { + searchService.search(serverCache, getSearchSettings(), selectedFunctions); + close(); + } + + @Override + protected void setServerCache(BSimServerCache serverCache) { + super.setServerCache(serverCache); + updateFilters(); + } + + protected void updateSearchFunctionsLabel() { + if (selectedFunctions.isEmpty()) { + functionsField.setText(""); + } + else if (selectedFunctions.size() == 1) { + FunctionSymbol symbol = selectedFunctions.iterator().next(); + functionsField.setText(symbol.getName()); + } + else { + functionsField.setText("" + selectedFunctions.size() + " selected functions"); + } + } + + protected JPanel buildServerPanel() { + JPanel panel = super.buildServerPanel(); + panel.add(new JLabel("Function(s): ")); + panel.add(buildSelectedFunctionPanel()); + return panel; + } + + protected JPanel buildCenterPanel() { + filterPanel = new BSimFilterPanel(this::filterPanelChanged); + return createTitledPanel("Filters:", filterPanel, true); + } + + @Override + protected boolean canQuery() { + if (!super.canQuery()) { + return false; + } + else if (!filterPanel.hasValidFilters()) { + setStatusText("One or more filters has invalid data!", MessageType.ERROR); + return false; + } + clearStatusText(); + return true; + } + + @Override + protected JPanel buildOptionsPanel() { + JPanel panel = super.buildOptionsPanel(); + + maxResultsField = new IntegerTextField(10); + maxResultsField.setValue(100); + maxResultsField.setMinValue(BigInteger.ONE); + maxResultsField.setAllowNegativeValues(false); + maxResultsField.setAllowsHexPrefix(false); + maxResultsField.setShowNumberMode(false); + + panel.add(new JLabel("Max Matches Per Function:")); + panel.add(maxResultsField.getComponent()); + return panel; + } + + protected BSimSearchSettings getSearchSettings() { + double similarity = similarityField.getValue(); + double confidence = confidenceField.getValue(); + int maxResults = maxResultsField.getIntValue(); + BSimFilterSet set = filterPanel.getFilterSet(); + return new BSimSearchSettings(similarity, confidence, maxResults, set); + } + + private void updateFilters() { + DatabaseInformation databaseInfo = getDatabaseInformation(); + List filters = BSimFilterType.generateBsimFilters(databaseInfo, true); + filterPanel.setFilters(filters); + } + + private void filterPanelChanged() { + updateSearchEnablement(); + } + + private Component buildSelectedFunctionPanel() { + JPanel panel = new JPanel(new BorderLayout()); + JPanel innerPanel = new JPanel(new BorderLayout()); + innerPanel.setBorder(BorderFactory.createEmptyBorder(0, 0, 0, 10)); + functionsField = new JTextField(20); + functionsField.setEditable(false); + innerPanel.add(functionsField, BorderLayout.CENTER); + panel.add(innerPanel, BorderLayout.CENTER); + JButton button = new EmptyBorderButton(FUNCTIONS_ICON); + button.setToolTipText("Show table of selected functions"); + button.addActionListener(e -> showSelectedFunctionsDialog()); + panel.add(button, BorderLayout.EAST); + return panel; + } + + private void showSelectedFunctionsDialog() { + if (selectedFunctions == null) { + return; + } + GoToService service = tool.getService(GoToService.class); + HelpLocation help = new HelpLocation("BSimSearchPlugin", "Selected_Functions"); + DialogComponentProvider dialog = + new SelectedFunctionsTableDialog(selectedFunctions, service, help); + DockingWindowManager.showDialog(dialog); + } + +//================================================================================================== +// Test methods +//================================================================================================== + Set getSelectedFunction() { + return selectedFunctions; + } + + BSimFilterPanel getFilterPanel() { + return filterPanel; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchService.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchService.java new file mode 100644 index 0000000000..cb2fe70694 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchService.java @@ -0,0 +1,57 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.util.Set; + +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.program.database.symbol.FunctionSymbol; + +/** + * Interface used by the BSimSearchDialog to initiate BSim Queries + */ +public interface BSimSearchService { + + /** + * Returns the BSimServerInfo that was used in the previous search or null if no searches + * have been performed. + * @return the BSimServerInfo that was used in the previous search + */ + public BSimServerInfo getLastUsedServer(); + + /** + * Returns the BSimSearchSettings that was used in the previous search or the default + * settings if no searches have been performed. + * @return the BSimSearchSettings that was used in the previous search + */ + public BSimSearchSettings getLastUsedSearchSettings(); + + /** + * Initiates a BSim similar functions search. + * @param severCache the server to query + * @param settings the settings to use for the search + * @param functions the functions to search for similar matches + */ + public void search(BSimServerCache severCache, BSimSearchSettings settings, + Set functions); + + /** + * Initiates a BSim overview search using all the functions in the program. + * @param severCache the server to query + * @param settings the settings to use for the search + */ + public void performOverview(BSimServerCache severCache, BSimSearchSettings settings); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchSettings.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchSettings.java new file mode 100644 index 0000000000..34c36e4957 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimSearchSettings.java @@ -0,0 +1,89 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +/** + * Class to hold all the settings for a BSim similar functions search + */ +public class BSimSearchSettings { + private double similarity; + private double confidence; + private int maxResults; + private BSimFilterSet filterSet; + + public BSimSearchSettings() { + this.similarity = 0.7; + this.confidence = 0.0; + this.maxResults = 100; + this.filterSet = new BSimFilterSet(); + } + + public BSimSearchSettings(double similarity, double confidence, int maxResults, + BSimFilterSet filterSet) { + this.similarity = similarity; + this.confidence = confidence; + this.maxResults = maxResults; + this.filterSet = filterSet; + } + + private BSimSearchSettings(BSimSearchSettings settings) { + this.similarity = settings.similarity; + this.confidence = settings.confidence; + this.maxResults = settings.getMaxResults(); + this.filterSet = settings.getBSimFilterSet().copy(); + } + + /** + * Returns the similarity criteria. + * @return the similarity criteria. + */ + public double getSimilarity() { + return similarity; + } + + /** + * Returns the confidence criteria. + * @return the confidence criteria. + */ + public double getConfidence() { + return confidence; + } + + /** + * Returns the maximum number of matches for a single function. + * @return the maximum number of matches for a single function + */ + public int getMaxResults() { + return maxResults; + } + + /** + * Returns the filters to be used for the query + * @return the filters to be used for the query + */ + public BSimFilterSet getBSimFilterSet() { + return filterSet; + } + + /** + * Returns a copy of this settings. + * @return a copy of this settings. + */ + public BSimSearchSettings copy() { + return new BSimSearchSettings(this); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerCache.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerCache.java new file mode 100644 index 0000000000..50010d40ec --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerCache.java @@ -0,0 +1,68 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.FunctionDatabase.ErrorCategory; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.facade.QueryDatabaseException; + +/** + * Caches BSim database info for a Bsim database connection + */ +public class BSimServerCache { + private BSimServerInfo serverInfo; + private DatabaseInformation databaseInfo; + private LSHVectorFactory lshVectorFactory; + + public BSimServerCache(BSimServerInfo severInfo) throws QueryDatabaseException { + this.serverInfo = severInfo; + initialize(); + } + + public BSimServerInfo getServerInfo() { + return serverInfo; + } + + public DatabaseInformation getDatabaseInformation() { + return databaseInfo; + } + + /** + * Get cached {@link LSHVectorFactory} for the active BSim Function Database + * @return vector factory or null if DB server not set or never connected + */ + public LSHVectorFactory getLSHVectorFactory() { + return lshVectorFactory; + } + + private void initialize() throws QueryDatabaseException { + try (FunctionDatabase database = serverInfo.getFunctionDatabase(false)) { + if (!database.initialize()) { // error message will be set on failure + String errorMessage = database.getLastError().message; + if (database.getLastError().category == ErrorCategory.Nodatabase) { + errorMessage = "Database does not exist"; + } + throw new QueryDatabaseException(errorMessage); + } + databaseInfo = database.getInfo(); + lshVectorFactory = database.getLSHVectorFactory(); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerDialog.java new file mode 100644 index 0000000000..a2c90f1c67 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerDialog.java @@ -0,0 +1,202 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.BorderLayout; + +import javax.swing.*; + +import org.bouncycastle.util.Arrays; + +import docking.DialogComponentProvider; +import docking.DockingWindowManager; +import docking.action.DockingAction; +import docking.action.builder.ActionBuilder; +import docking.widgets.OptionDialog; +import docking.widgets.PasswordChangeDialog; +import docking.widgets.table.GFilterTable; +import docking.widgets.table.GTable; +import generic.theme.GIcon; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.FunctionDatabase.Error; +import ghidra.features.bsim.query.FunctionDatabase.ErrorCategory; +import ghidra.framework.plugintool.PluginTool; +import ghidra.util.*; +import resources.Icons; + +/** + * Dialog for managing BSim database server definitions + */ +public class BSimServerDialog extends DialogComponentProvider { + + // TODO: Add connected status indicator (not sure how this relates to elastic case which will likely have a Session concept) + // TODO: Add "Disconnect" action (only works when active connections is 0; does not apply to elastic) + + private PluginTool tool; + private BSimServerManager serverManager; + private BSimServerTableModel serverTableModel; + private GFilterTable filterTable; + private BSimServerInfo lastAdded = null; + + public BSimServerDialog(PluginTool tool, BSimServerManager serverManager) { + super("BSim Server Manager"); + this.tool = tool; + this.serverManager = serverManager; + addWorkPanel(buildMainPanel()); + createToolbarActions(); + addDismissButton(); + setPreferredSize(600, 400); + notifyContextChanged(); // kick actions to initialized enabled state + setHelpLocation(new HelpLocation("BSimSearchPlugin","BSim_Servers_Dialog" )); + } + + @Override + protected void dismissCallback() { + serverTableModel.dispose(); + super.dismissCallback(); + } + + private void createToolbarActions() { + HelpLocation help = new HelpLocation("BSimSearchPlugin","Manage_Servers_Actions" ); + + DockingAction addServerAction = + new ActionBuilder("Add Server", "Dialog").toolBarIcon(Icons.ADD_ICON) + .helpLocation(help) + .onAction(e -> defineBsimServer()) + .build(); + addAction(addServerAction); + DockingAction removeServerAction = + new ActionBuilder("Delete Server", "Dialog").toolBarIcon(Icons.DELETE_ICON) + .helpLocation(help) + .onAction(e -> deleteBsimServer()) + .enabledWhen(c -> hasSelection()) + .build(); + addAction(removeServerAction); + + DockingAction changePasswordAction = new ActionBuilder("Change User Password", "Dialog") + .helpLocation(help) + .toolBarIcon(new GIcon("icon.bsim.change.password")) + .onAction(e -> changePassword()) + .enabledWhen(c -> hasSelection()) + .build(); + addAction(changePasswordAction); + + } + + private void changePassword() { + BSimServerInfo serverInfo = filterTable.getSelectedRowObject(); + if (serverInfo == null) { + return; + } + char[] pwd = null; + try (FunctionDatabase db = BSimClientFactory.buildClient(serverInfo, true)) { + if (!db.initialize()) { + // TODO: Need standardized error handler + Error lastError = db.getLastError(); + if (lastError.category != ErrorCategory.AuthenticationCancelled) { + Msg.showError(this, getComponent(), "BSim DB Connection Failed", + lastError.message); + } + return; + } + if (!db.isPasswordChangeAllowed()) { + Msg.showError(this, getComponent(), "Unsupported Operation", + "BSim DB password change not supported"); + return; + } + PasswordChangeDialog dlg = new PasswordChangeDialog("Change Password", "BSim DB", + serverInfo.toString(), db.getUserName()); + tool.showDialog(dlg); + pwd = dlg.getPassword(); + if (pwd == null) { + return; // password dialog entry cancelled by user + } + + String resp = db.changePassword(db.getUserName(), pwd); + if (resp == null) { + Msg.showInfo(this, getComponent(), "Password Changed", + "BSim DB password successfully changed"); + } + else { + Msg.showError(this, getComponent(), "Password Changed Failed", resp); + } + } + finally { + if (pwd != null) { + Arrays.fill(pwd, '\0'); + } + } + } + + private void deleteBsimServer() { + BSimServerInfo selected = filterTable.getSelectedRowObject(); + if (selected != null) { + int answer = + OptionDialog.showYesNoDialog(getComponent(), "Delete Server Configuration?", + "Are you sure you want to delete: " + selected + "?"); + if (answer == OptionDialog.YES_OPTION) { + if (!serverManager.removeServer(selected, false)) { + answer = OptionDialog.showOptionDialogWithCancelAsDefaultButton(getComponent(), + "Active Server Configuration!", + "Database connections are still active!\n" + + "Are you sure you want to delete server?", + "Yes", OptionDialog.WARNING_MESSAGE); + if (answer == OptionDialog.YES_OPTION) { + serverManager.removeServer(selected, true); + } + } + } + } + } + + private void defineBsimServer() { + CreateBsimServerInfoDialog dialog = new CreateBsimServerInfoDialog(); + DockingWindowManager.showDialog(dialog); + BSimServerInfo newServerInfo = dialog.getBsimServerInfo(); + if (newServerInfo != null) { + serverManager.addServer(newServerInfo); + lastAdded = newServerInfo; + Swing.runLater(() -> filterTable.setSelectedRowObject(newServerInfo)); + } + } + + private JComponent buildMainPanel() { + JPanel panel = new JPanel(new BorderLayout()); + panel.setBorder(BorderFactory.createEmptyBorder(10, 10, 10, 10)); + + serverTableModel = new BSimServerTableModel(serverManager); + filterTable = new GFilterTable<>(serverTableModel); + GTable table = filterTable.getTable(); + table.setSelectionMode(ListSelectionModel.SINGLE_SELECTION); + table.getSelectionModel().addListSelectionListener(e -> notifyContextChanged()); + panel.add(filterTable, BorderLayout.CENTER); + + if (serverTableModel.getRowCount() > 0) { + table.setRowSelectionInterval(0, 0); + } + + return panel; + } + + private boolean hasSelection() { + return filterTable.getSelectedRowObject() != null; + } + + public BSimServerInfo getLastAdded() { + return lastAdded; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerManager.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerManager.java new file mode 100644 index 0000000000..9e1559af07 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerManager.java @@ -0,0 +1,185 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.io.File; +import java.io.IOException; +import java.util.*; +import java.util.concurrent.CopyOnWriteArrayList; + +import ghidra.features.bsim.query.BSimPostgresDBConnectionManager; +import ghidra.features.bsim.query.BSimPostgresDBConnectionManager.BSimPostgresDataSource; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager.BSimH2FileDataSource; +import ghidra.framework.Application; +import ghidra.framework.options.GProperties; +import ghidra.framework.options.JSonProperties; +import ghidra.util.Msg; +import ghidra.util.Swing; + +/** + * Managers BSim database server definitions and connections + */ +public class BSimServerManager { + // TODO: Do not allow removal of active server. Dispose data source when removed. + + private Set serverInfos = new HashSet<>(); + private List listeners = new CopyOnWriteArrayList<>(); + + public BSimServerManager() { + List files = Application.getUserSettingsFiles("bsim", ".server.properties"); + for (File file : files) { + BSimServerInfo info = readBsimServerInfoFile(file); + if (info != null) { + serverInfos.add(info); + } + } + } + + public Set getServerInfos() { + return new HashSet<>(serverInfos); + } + + private BSimServerInfo readBsimServerInfoFile(File file) { + try { + GProperties properties = new JSonProperties(file); + String dbTypeName = properties.getString("DBType", null); + DBType dbType = DBType.valueOf(dbTypeName); + String name = properties.getString("Name", null); + String host = properties.getString("Host", null); + int port = properties.getInt("Port", 0); + if (dbType != null && name != null) { + BSimServerInfo info = new BSimServerInfo(dbType, host, port, name); + return info; + } + Msg.showError(this, null, "Error reading Bsim Server File", + "Bad BSim Server Info in file " + file.toString()); + } + catch (IOException e) { + Msg.showError(this, null, "Error Reading BSim Server File", + "Error processing Bsim Server info file: " + file.toString()); + } + return null; + } + + private boolean saveBSimServerInfo(BSimServerInfo info) { + GProperties properties = new GProperties("BSimServerInfo"); + properties.putString("DBType", info.getDBType().name()); + properties.putString("Name", info.getDBName()); + properties.putString("Host", info.getServerName()); + properties.putInt("Port", info.getPort()); + + File settingsDir = Application.getUserSettingsDirectory(); + File serversDir = new File(settingsDir, "bsim"); + int hash = info.hashCode(); + File serverFile = new File(serversDir, "bsim" + hash + ".server.properties"); + try { + properties.saveToJsonFile(serverFile); + return true; + } + catch (IOException e) { + Msg.showError(this, null, "Error Saving", + "Error saving Bsim Server Info to file " + serverFile.getAbsolutePath(), e); + return false; + } + + } + + private boolean removeServerFileFromSettings(BSimServerInfo info) { + File settingsDir = Application.getUserSettingsDirectory(); + File serversDir = new File(settingsDir, "bsim"); + int hash = info.hashCode(); + File serverFile = new File(serversDir, "bsim" + hash + ".server.properties"); + return serverFile.delete(); + } + + public void addServer(BSimServerInfo newServerInfo) { + if (saveBSimServerInfo(newServerInfo)) { + serverInfos.add(newServerInfo); + notifyServerListChanged(); + } + } + + public boolean removeServer(BSimServerInfo info, boolean force) { + DBType dbType = info.getDBType(); + if (dbType == DBType.file) { + BSimH2FileDataSource ds = BSimH2FileDBConnectionManager.getDataSource(info); + int active = ds.getActiveConnections(); + if (active != 0) { + if (!force) { + return false; + } + ds.dispose(); + } + } + else if (dbType == DBType.postgres) { + BSimPostgresDataSource ds = BSimPostgresDBConnectionManager.getDataSource(info); + int active = ds.getActiveConnections(); + if (active != 0) { + if (!force) { + return false; + } + ds.dispose(); + } + } + if (serverInfos.remove(info)) { + removeServerFileFromSettings(info); + notifyServerListChanged(); + } + return true; + } + + public void addListener(BSimServerManagerListener listener) { + listeners.add(listener); + } + + public void removeListener(BSimServerManagerListener listener) { + listeners.remove(listener); + } + + private void notifyServerListChanged() { + Swing.runLater(() -> { + for (BSimServerManagerListener listener : listeners) { + listener.serverListChanged(); + } + }); + } + + public static int getActiveConnections(BSimServerInfo serverInfo) { + switch (serverInfo.getDBType()) { + case postgres: + BSimPostgresDataSource postgresDs = + BSimPostgresDBConnectionManager.getDataSourceIfExists(serverInfo); + if (postgresDs != null) { + return postgresDs.getActiveConnections(); + } + break; + case file: + BSimH2FileDataSource h2FileDs = + BSimH2FileDBConnectionManager.getDataSourceIfExists(serverInfo); + if (h2FileDs != null) { + return h2FileDs.getActiveConnections(); + } + break; + default: + break; + } + return -1; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerManagerListener.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerManagerListener.java new file mode 100644 index 0000000000..0a77f5262b --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerManagerListener.java @@ -0,0 +1,23 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +/** + * Listener for when the list of defined BSim database definitions change + */ +public interface BSimServerManagerListener { + public void serverListChanged(); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerTableModel.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerTableModel.java new file mode 100644 index 0000000000..b26aeea431 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/BSimServerTableModel.java @@ -0,0 +1,235 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.Component; +import java.util.ArrayList; +import java.util.List; + +import javax.swing.JLabel; + +import docking.widgets.table.*; +import ghidra.docking.settings.Settings; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.framework.plugintool.ServiceProviderStub; +import ghidra.program.model.listing.Program; +import ghidra.util.table.column.AbstractGColumnRenderer; +import ghidra.util.table.column.GColumnRenderer; +import ghidra.util.table.field.AbstractProgramBasedDynamicTableColumn; + +/** + * Table model for BSim database server definitions + */ +public class BSimServerTableModel extends GDynamicColumnTableModel { + private List servers; + private BSimServerManager serverManager; + private BSimServerManagerListener listener = new BSimServerManagerListener() { + @Override + public void serverListChanged() { + updateServers(); + } + }; + + public BSimServerTableModel(BSimServerManager serverManager) { + super(new ServiceProviderStub()); + this.serverManager = serverManager; + serverManager.addListener(listener); + servers = new ArrayList<>(serverManager.getServerInfos()); + } + + @Override + public String getName() { + return "BSim Servers"; + } + + @Override + public List getModelData() { + return servers; + } + + @Override + public boolean isSortable(int columnIndex) { + return columnIndex != 0; + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = new TableColumnDescriptor<>(); + descriptor.addVisibleColumn(new DatabaseNameColumn(), 1, true); + descriptor.addVisibleColumn(new TypeColumn()); + descriptor.addVisibleColumn(new HostColumn()); + descriptor.addVisibleColumn(new PortColumn()); + descriptor.addVisibleColumn(new ActiveConnectionColumn()); + return descriptor; + } + + @Override + public Object getDataSource() { + return null; + } + + private class DatabaseNameColumn + extends AbstractProgramBasedDynamicTableColumn { + private GColumnRenderer renderer = new AbstractGColumnRenderer<>() { + @Override + public Component getTableCellRendererComponent(GTableCellRenderingData data) { + JLabel label = (JLabel) super.getTableCellRendererComponent(data); + BSimServerInfo info = (BSimServerInfo) data.getRowObject(); + if (info.getDBType() == DBType.file) { + label.setToolTipText(info.getDBName()); + } + else { + label.setToolTipText(""); + } + return label; + } + + @Override + public String getFilterString(String value, Settings settings) { + return value; + } + + }; + + @Override + public String getColumnName() { + return "Name"; + } + + @Override + public String getValue(BSimServerInfo serverInfo, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + + // FIXME: Get cell tooltip to show full getDBName which includes file path + + return serverInfo.getShortDBName(); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + + @Override + public GColumnRenderer getColumnRenderer() { + return renderer; + } + } + + private class HostColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Host"; + } + + @Override + public String getValue(BSimServerInfo serverInfo, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + + return serverInfo.getServerName(); + } + + @Override + public int getColumnPreferredWidth() { + return 150; + } + } + + private class PortColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Port"; + } + + @Override + public Integer getValue(BSimServerInfo serverInfo, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + + int port = serverInfo.getPort(); + if (port <= 0) { + return null; + } + return port; + } + + @Override + public int getColumnPreferredWidth() { + return 80; + } + } + + private class ActiveConnectionColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Active Connections"; + } + + @Override + public Integer getValue(BSimServerInfo serverInfo, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + int activeConnections = BSimServerManager.getActiveConnections(serverInfo); + if (activeConnections < 0) { + return null; + } + return activeConnections; + } + + @Override + public int getColumnPreferredWidth() { + return 80; + } + } + + private class TypeColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Type"; + } + + @Override + public String getValue(BSimServerInfo serverInfo, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + + return serverInfo.getDBType().toString(); + } + + @Override + public int getColumnPreferredWidth() { + return 80; + } + } + + private void updateServers() { + servers = new ArrayList<>(serverManager.getServerInfos()); + fireTableDataChanged(); + } + + @Override + public void dispose() { + serverManager.removeListener(listener); + super.dispose(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/CreateBsimServerInfoDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/CreateBsimServerInfoDialog.java new file mode 100644 index 0000000000..61adf07ce1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/CreateBsimServerInfoDialog.java @@ -0,0 +1,349 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.*; +import java.awt.event.ActionListener; +import java.io.File; + +import javax.swing.*; +import javax.swing.event.DocumentEvent; +import javax.swing.event.DocumentListener; + +import docking.DialogComponentProvider; +import docking.widgets.OptionDialog; +import docking.widgets.button.BrowseButton; +import docking.widgets.button.GRadioButton; +import docking.widgets.filechooser.GhidraFileChooser; +import docking.widgets.filechooser.GhidraFileChooserMode; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.util.HelpLocation; +import ghidra.util.filechooser.GhidraFileChooserModel; +import ghidra.util.filechooser.GhidraFileFilter; +import ghidra.util.layout.*; + +/** + * Dialog for entering new BSim database server definition + */ +public class CreateBsimServerInfoDialog extends DialogComponentProvider { + protected static final String FILE_DB_EXT = ".mv.db"; + + private static final String POSTGRES = "Postgres"; + private static final String ELASTIC = "Elastic"; + private static final String FILE_H2 = "File"; + private GRadioButton postgresButton; + private GRadioButton elasticButton; + private GRadioButton fileButton; + + private JPanel cardPanel; + + private PostgresPanel postgresPanel; + private ElasticPanel elasticPanel; + private FilePanel filePanel; + + private ServerPanel activePanel; + private BSimServerInfo result; + + public CreateBsimServerInfoDialog() { + super("Add BSim Server"); + addWorkPanel(buildMainPanel()); + + addOKButton(); + addCancelButton(); + setOkEnabled(false); + setHelpLocation(new HelpLocation("BSimSearchPlugin","Add_Server_Definition_Dialog" )); + } + + public BSimServerInfo getBsimServerInfo() { + return result; + } + @Override + public void setHelpLocation(HelpLocation helpLocation) { + // TODO Auto-generated method stub + super.setHelpLocation(helpLocation); + } + + @Override + protected void okCallback() { + BSimServerInfo serverInfo = activePanel.getServerInfo(); + // FIXME: serverInfo may be null - seems like OK button should have been disabled + if (acceptServer(serverInfo)) { + result = serverInfo; + close(); + } + } + + public boolean acceptServer(BSimServerInfo serverInfo) { + // FIXME: Use task to correct dialog parenting issue caused by password prompt + String errorMessage = null; + try { + FunctionDatabase database = BSimClientFactory.buildClient(serverInfo, true); + if (database.initialize()) { + return true; + } + errorMessage = database.getLastError().toString(); + } + catch (Exception e) { + errorMessage = e.getMessage(); + } + int answer = OptionDialog.showYesNoDialog(null, "Connection Test Failed!", + "Can't connect to server: " + errorMessage + "\nDo you want create anyway?"); + return answer == OptionDialog.YES_OPTION; + } + + private JComponent buildMainPanel() { + JPanel panel = new JPanel(new BorderLayout()); + panel.add(buildTypePanel(), BorderLayout.NORTH); + panel.add(buildCardPanel(), BorderLayout.CENTER); + return panel; + } + + private Component buildCardPanel() { + postgresPanel = new PostgresPanel(); + elasticPanel = new ElasticPanel(); + filePanel = new FilePanel(); + + cardPanel = new JPanel(new CardLayout()); + cardPanel.add(postgresPanel, POSTGRES); + cardPanel.add(elasticPanel, ELASTIC); + cardPanel.add(filePanel, FILE_H2); + activePanel = postgresPanel; + return cardPanel; + } + + private Component buildTypePanel() { + JPanel panel = new JPanel(new MiddleLayout()); + panel.setBorder(BorderFactory.createEmptyBorder(20, 10, 10, 10)); + JPanel innerPanel = new JPanel(new HorizontalLayout(20)); + ButtonGroup group = new ButtonGroup(); + postgresButton = new GRadioButton(POSTGRES); + elasticButton = new GRadioButton(ELASTIC); + fileButton = new GRadioButton(FILE_H2); + + postgresButton.setSelected(true); + + ActionListener actionListener = e -> radioChanged(); + postgresButton.addActionListener(actionListener); + elasticButton.addActionListener(actionListener); + fileButton.addActionListener(actionListener); + + group.add(postgresButton); + group.add(elasticButton); + group.add(fileButton); + + innerPanel.add(postgresButton); + innerPanel.add(elasticButton); + innerPanel.add(fileButton); + panel.add(innerPanel); + + return panel; + } + + private void radioChanged() { + CardLayout cardLayout = (CardLayout) cardPanel.getLayout(); + if (postgresButton.isSelected()) { + cardLayout.show(cardPanel, POSTGRES); + activePanel = postgresPanel; + } + else if (elasticButton.isSelected()) { + cardLayout.show(cardPanel, ELASTIC); + activePanel = elasticPanel; + } + else if (fileButton.isSelected()) { + cardLayout.show(cardPanel, FILE_H2); + activePanel = filePanel; + } + checkForValidDialog(); + } + + private int getPort(String portString) { + try { + return Integer.parseInt(portString); + } + catch (Throwable t) { + return -1; + } + } + + private abstract class ServerPanel extends JPanel { + ServerPanel(LayoutManager layout) { + super(layout); + setBorder(BorderFactory.createEmptyBorder(20, 10, 10, 10)); + } + + abstract BSimServerInfo getServerInfo(); + } + + private void checkForValidDialog() { + BSimServerInfo serverInfo = activePanel.getServerInfo(); + setOkEnabled(serverInfo != null); + } + + private class PostgresPanel extends ServerPanel { + + private JTextField nameField; + private JTextField hostField; + private JTextField portField; + + PostgresPanel() { + super(new PairLayout(10, 10)); + nameField = new NotifyingTextField(); + hostField = new NotifyingTextField(); + portField = + new NotifyingTextField(Integer.toString(BSimServerInfo.DEFAULT_POSTGRES_PORT)); + add(new JLabel("DB Name:", SwingConstants.RIGHT)); + add(nameField); + add(new JLabel("Host:", SwingConstants.RIGHT)); + add(hostField); + add(new JLabel("Port:", SwingConstants.RIGHT)); + add(portField); + } + + @Override + BSimServerInfo getServerInfo() { + String name = nameField.getText().trim(); + String host = hostField.getText().trim(); + int port = getPort(portField.getText().trim()); + if (name.isBlank() || host.isBlank() || port < 0) { + return null; + } + return new BSimServerInfo(DBType.postgres, host, port, name); + } + } + + private class ElasticPanel extends ServerPanel { + + private JTextField nameField; + private JTextField hostField; + private JTextField portField; + + ElasticPanel() { + super(new PairLayout(10, 10)); + nameField = new NotifyingTextField(); + hostField = new NotifyingTextField(); + portField = + new NotifyingTextField(Integer.toString(BSimServerInfo.DEFAULT_ELASTIC_PORT)); + add(new JLabel("DB Name:", SwingConstants.RIGHT)); + add(nameField); + add(new JLabel("Host:", SwingConstants.RIGHT)); + add(hostField); + add(new JLabel("Port:", SwingConstants.RIGHT)); + add(portField); + } + + @Override + BSimServerInfo getServerInfo() { + String name = nameField.getText().trim(); + String host = hostField.getText().trim(); + int port = getPort(portField.getText().trim()); + if (name.isBlank() || host.isBlank() || port < 0) { + return null; + } + return new BSimServerInfo(DBType.elastic, host, port, name); + } + + } + + private class FilePanel extends ServerPanel { + private JTextField fileField; + + FilePanel() { + super(new PairLayout()); + add(new JLabel("File: ")); + add(buildFileField()); + } + + private JPanel buildFileField() { + JPanel panel = new JPanel(new BorderLayout()); + fileField = new NotifyingTextField(); + fileField.setEditable(false); + panel.add(fileField, BorderLayout.CENTER); + JPanel subpanel = new JPanel(new MiddleLayout()); + subpanel.setBorder(BorderFactory.createEmptyBorder(0, 10, 0, 0)); + BrowseButton browseButton = new BrowseButton(); + browseButton.addActionListener(e -> showFileChooser()); + subpanel.add(browseButton); + panel.add(subpanel, BorderLayout.EAST); + return panel; + } + + private void showFileChooser() { + GhidraFileChooser chooser = new GhidraFileChooser(this); + chooser.setFileSelectionMode(GhidraFileChooserMode.FILES_ONLY); + chooser.setFileFilter(new GhidraFileFilter() { + + @Override + public String getDescription() { + return "*" + FILE_DB_EXT; + } + + @Override + public boolean accept(File file, GhidraFileChooserModel chooserModel) { + return file.isDirectory() || file.getName().endsWith(FILE_DB_EXT); + } + }); + + File selectedFile = chooser.getSelectedFile(); + if (selectedFile != null) { + fileField.setText(selectedFile.getAbsolutePath()); + } + } + + @Override + BSimServerInfo getServerInfo() { + String path = fileField.getText().trim(); + if (path.isBlank()) { + return null; + } + File file = new File(path); + if (file.isDirectory()) { + return null; + } + return new BSimServerInfo(DBType.file, null, -1, path); + } + } + + class NotifyingTextField extends JTextField { + public NotifyingTextField() { + this(""); + } + + public NotifyingTextField(String initialText) { + super(20); + setText(initialText); + getDocument().addDocumentListener(new DocumentListener() { + + @Override + public void insertUpdate(DocumentEvent e) { + checkForValidDialog(); + } + + @Override + public void removeUpdate(DocumentEvent e) { + checkForValidDialog(); + } + + @Override + public void changedUpdate(DocumentEvent e) { + checkForValidDialog(); + } + + }); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/FilterWidget.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/FilterWidget.java new file mode 100755 index 0000000000..05e9a226dd --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/FilterWidget.java @@ -0,0 +1,193 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.BorderLayout; +import java.awt.Component; +import java.awt.event.ItemEvent; +import java.awt.event.ItemListener; +import java.util.ArrayList; +import java.util.List; +import java.util.function.Consumer; + +import javax.swing.*; + +import docking.widgets.EmptyBorderButton; +import docking.widgets.combobox.GhidraComboBox; +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.gui.filters.BSimValueEditor; +import ghidra.util.Swing; +import resources.Icons; +import utility.function.Callback; + +/** + * This class defines a widget for a single BSim filter. At a minimum + * it will consist of a combobox containing the available filters. It may optionally + * contain a secondary widget for specifying filter values. This secondary widget + * is filter-specific; for most filter types it will be a text entry field but as long + * as it implements the {@link FilterContent} interface it is valid. + * + */ +public class FilterWidget extends JPanel { + private GhidraComboBox filterComboBox; + private JPanel contentPanel; + + private BSimFilterType filterType; + private BSimValueEditor editor; + + private ItemListener comboChangeListener = this::comboChanged; + private Callback changeListener; + + /** + * Constructs a new filter widget. + * @param filterTypes The list of filter types that can be chosen + * @param removeConsumer the container to be notified that it should delete this object + * @param renameListener listener to be notified when filter value changes + */ + public FilterWidget(List filterTypes, Consumer removeConsumer, + Callback changeListener) { + + this.changeListener = changeListener; + this.filterType = filterTypes.get(0); // the first is the blank filter + + setLayout(new BorderLayout()); + add(createFilterComboBox(filterTypes), BorderLayout.WEST); + add(buildFilterContentPanel(), BorderLayout.CENTER); + add(buildDeleteButton(removeConsumer), BorderLayout.EAST); + + } + + public void setFilters(List filters) { + if (filters == null || filters.isEmpty()) { + filters = List.of(BSimFilterType.BLANK); + } + + filterComboBox.removeItemListener(comboChangeListener); + + DefaultComboBoxModel model = new DefaultComboBoxModel<>(); + model.addAll(filters); + filterComboBox.setModel(model); + + if (filterType == null || !filters.contains(filterType)) { + filterType = filters.get(0); + createInputField(filterType); + } + filterComboBox.setSelectedItem(filterType); + + filterComboBox.addItemListener(comboChangeListener); + } + + public void setFilter(BSimFilterType filter, List values) { + ComboBoxModel model = filterComboBox.getModel(); + for (int i = 0; i < model.getSize(); i++) { + BSimFilterType newFilterType = model.getElementAt(i); + if (filter.equals(newFilterType)) { + filterComboBox.setSelectedIndex(i); + editor.setValues(values); + break; + } + } + } + + /** + * Returns the selected filter. + * @return the filter + */ + public BSimFilterType getSelectedFilter() { + return filterType; + } + + /** + * Returns all values in the filter as a list. For filters that do not allow + * multiple entries, this will always return a list of only one item. + * + * @return filter values + * + */ + public List getValues() { + return editor.getValues(); + } + + private void createInputField(BSimFilterType filter) { + contentPanel.removeAll(); + editor = createEditor(filter, null); + contentPanel.add(editor.getComponent(), BorderLayout.CENTER); + revalidate(); + } + + private BSimValueEditor createEditor(BSimFilterType filter, List initialValues) { + if (filter == null) { + return null; + } + return filter.getEditor(initialValues, changeListener); + } + + private GhidraComboBox createFilterComboBox(List filters) { + filterComboBox = new GhidraComboBox<>(); + setFilters(filters); + return filterComboBox; + } + + private Component buildDeleteButton(Consumer removeConsumer) { + JButton deleteButton = new EmptyBorderButton(Icons.DELETE_ICON); + deleteButton.setToolTipText("Delete filter"); + deleteButton.setName("Delete Filter"); + deleteButton.addActionListener(e -> { + removeConsumer.accept(this); + }); + return deleteButton; + } + + private Component buildFilterContentPanel() { + editor = createEditor(filterType, null); + contentPanel = new JPanel(new BorderLayout()); + contentPanel.setBorder(BorderFactory.createEmptyBorder(0, 10, 0, 10)); + contentPanel.add(editor.getComponent()); + return contentPanel; + } + + private void comboChanged(ItemEvent e) { + if (e.getStateChange() == ItemEvent.SELECTED) { + filterType = (BSimFilterType) e.getItem(); + createInputField(filterType); + notifyChangeListener(); + } + } + + private void notifyChangeListener() { + Swing.runLater(() -> changeListener.call()); + } + + public boolean isBlank() { + return filterType.isBlank(); + } + + public boolean hasValidValue() { + return editor.hasValidValues(); + } +//================================================================================================== +// Test methods +//================================================================================================== + + List getChoosableFilterTypes() { + ComboBoxModel model = filterComboBox.getModel(); + List types = new ArrayList<>(); + for (int i = 0; i < model.getSize(); i++) { + types.add(model.getElementAt(i)); + } + return types; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/FunctionSymbolToFunctionTableRowMapper.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/FunctionSymbolToFunctionTableRowMapper.java new file mode 100644 index 0000000000..f64af2a00c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/FunctionSymbolToFunctionTableRowMapper.java @@ -0,0 +1,35 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.util.table.ProgramLocationTableRowMapper; + +/** + * Maps FunctionSymbols to Functions to get table columns for functions + */ +public class FunctionSymbolToFunctionTableRowMapper + extends ProgramLocationTableRowMapper { + + @Override + public Function map(FunctionSymbol rowObject, Program data, ServiceProvider serviceProvider) { + return (Function) rowObject.getObject(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/SelectedFunctionsTableDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/SelectedFunctionsTableDialog.java new file mode 100644 index 0000000000..0dc804c7e4 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/dialog/SelectedFunctionsTableDialog.java @@ -0,0 +1,190 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.awt.BorderLayout; +import java.util.Map; +import java.util.Set; + +import javax.swing.*; + +import docking.DialogComponentProvider; +import docking.widgets.table.DiscoverableTableUtils; +import docking.widgets.table.TableColumnDescriptor; +import ghidra.app.services.GoToService; +import ghidra.docking.settings.Settings; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.framework.plugintool.ServiceProviderStub; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Program; +import ghidra.util.HelpLocation; +import ghidra.util.datastruct.Accumulator; +import ghidra.util.exception.CancelledException; +import ghidra.util.table.AddressBasedTableModel; +import ghidra.util.table.GhidraFilterTable; +import ghidra.util.table.field.AbstractProgramBasedDynamicTableColumn; +import ghidra.util.table.field.FunctionSignatureTableColumn; +import ghidra.util.task.TaskMonitor; + +/** + * Dialog for display selected functions + */ +public class SelectedFunctionsTableDialog extends DialogComponentProvider { + + private Set functions; + private FunctionsTableModel model; + private Map matchCounts; + + public SelectedFunctionsTableDialog(Set functionSymbols, + GoToService gotoService, HelpLocation help) { + this(functionSymbols, gotoService, help, null); + } + + public SelectedFunctionsTableDialog(Set functionSymbols, + GoToService gotoService, HelpLocation help, Map matchCounts) { + super("Selected Functions For Bsim Search"); + this.functions = functionSymbols; + this.matchCounts = matchCounts; + addWorkPanel(buildMainPanel(gotoService)); + addDismissButton(); + setHelpLocation(help); + } + + private Program getProgram() { + if (functions.isEmpty()) { + return null; + } + FunctionSymbol next = functions.iterator().next(); + return next.getProgram(); + } + + private JComponent buildMainPanel(GoToService goToService) { + JPanel panel = new JPanel(new BorderLayout()); + panel.setBorder(BorderFactory.createEmptyBorder(10, 10, 10, 10)); + panel.add(buildFunctionsTable(getProgram(), goToService), BorderLayout.CENTER); + return panel; + } + + private JComponent buildFunctionsTable(Program program, GoToService goToService) { + model = new FunctionsTableModel(program); + GhidraFilterTable table = new GhidraFilterTable<>(model); + table.setNavigateOnSelectionEnabled(true); + table.installNavigation(goToService); + return table; + } + + private class FunctionsTableModel extends AddressBasedTableModel { + + public FunctionsTableModel(Program program) { + super("Selected Functions", new ServiceProviderStub(), program, TaskMonitor.DUMMY); + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = new TableColumnDescriptor<>(); + descriptor.addVisibleColumn(new SymbolNameColumn()); + + descriptor.addVisibleColumn(new SymbolAddressColumn(), 1, true); + + descriptor.addVisibleColumn(DiscoverableTableUtils.adaptColumForModel(this, + new FunctionSignatureTableColumn())); + + if (matchCounts != null) { + descriptor.addVisibleColumn(new MatchCountColumn()); + } + + return descriptor; + } + + @Override + public Address getAddress(int row) { + return getModelData().get(row).getAddress(); + } + + @Override + protected void doLoad(Accumulator accumulator, TaskMonitor monitor) + throws CancelledException { + accumulator.addAll(functions); + } + + private class SymbolNameColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Name"; + } + + @Override + public String getValue(FunctionSymbol symbol, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + + return symbol.getName(); + } + + @Override + public int getColumnPreferredWidth() { + return 150; + } + } + + private class MatchCountColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Matches"; + } + + @Override + public Integer getValue(FunctionSymbol symbol, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + + Integer count = matchCounts.get(symbol.getAddress()); + return count != null ? count : 0; + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + } + + private class SymbolAddressColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Address"; + } + + @Override + public String getValue(FunctionSymbol symbol, Settings settings, Program data, + ServiceProvider provider) throws IllegalArgumentException { + Address addr = symbol.getAddress(); + return addr.toString(); + } + + @Override + public int getColumnPreferredWidth() { + return 150; + } + } + + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResult.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResult.java new file mode 100755 index 0000000000..a504b0b609 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResult.java @@ -0,0 +1,97 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; + +/** + * Contains information regarding the result of a BSim 'apply function name' operation. It + * indicates the function name being changed, the new name to use, the address, and any + * pertinent error/informational text. + * + */ +public class BSimApplyResult { + + private String target; + private String source; + private BSimResultStatus status; + private Address address; + private String message; + + public BSimApplyResult(BSimMatchResult result, BSimResultStatus status, String message) { + this(result.getOriginalFunctionName(), result.getSimilarFunctionName(), status, + result.getAddress(), message); + } + + public BSimApplyResult(Function target, Function source, BSimResultStatus status, + String message) { + this(target.getName(true), source.getName(true), status, target.getEntryPoint(), message); + } + + public BSimApplyResult(String target, String source, BSimResultStatus status, Address address, + String message) { + this.target = target; + this.source = source; + this.status = status; + this.address = address; + this.message = message; + } + + /** + * @return the target function name + */ + public String getTargetFunctionName() { + return target; + } + + /** + * @return the similar function name + */ + public String getSourceFunctionName() { + return source; + } + + /** + * @return the status + */ + public BSimResultStatus getStatus() { + return status; + } + + /** + * @return the address + */ + public Address getAddress() { + return address; + } + + /** + * @return the message + */ + public String getMessage() { + return message; + } + + public boolean isError() { + return status == BSimResultStatus.ERROR; + } + + public boolean isIgnored() { + return status == BSimResultStatus.IGNORED; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResultsDisplayDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResultsDisplayDialog.java new file mode 100755 index 0000000000..9dc2c67556 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResultsDisplayDialog.java @@ -0,0 +1,120 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.awt.BorderLayout; +import java.awt.Dimension; +import java.util.List; + +import javax.swing.JPanel; + +import docking.DialogComponentProvider; +import docking.widgets.table.GTableFilterPanel; +import ghidra.features.bsim.gui.search.results.apply.AbstractBSimApplyTask; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.listing.Program; +import ghidra.util.table.GhidraFilterTable; +import ghidra.util.table.GhidraTable; + +/** + * Panel that displays the results of executing an "apply-rename" operation on a set of BSim + * query results. The results are presented in a {@link GhidraTable}; see the + * {@link BSimApplyResultsTableModel} for details on its structure. + *

+ * Filtering is provided on the results using a standard {@link GTableFilterPanel}. + * + * @see BSimApplyResultsTableModel + * @see AbstractBSimApplyTask + * + */ +public class BSimApplyResultsDisplayDialog extends DialogComponentProvider { + + private ServiceProvider serviceProvider; + private Program program; + private GhidraFilterTable table; + + public BSimApplyResultsDisplayDialog(ServiceProvider serviceProvider, + List results, Program program) { + super(createTitle(results)); + setRememberSize(false); + + this.serviceProvider = serviceProvider; + this.program = program; + + addWorkPanel(createWorkPanel(results)); + addOKButton(); + + } + + private static String createTitle(List results) { + StringBuilder builder = new StringBuilder(); + int successes = 0; + int errors = 0; + int ignored = 0; + for (BSimApplyResult result : results) { + if (result.isError()) { + errors++; + } + else if (result.isIgnored()) { + ignored++; + } + else { + successes++; + } + } + + builder.append("BSim Apply Results ("); + builder.append(successes); + builder.append(" successfully applied"); + if (errors > 0) { + builder.append(", "); + builder.append(errors); + builder.append(" error(s)"); + } + if (ignored > 0) { + builder.append(", "); + builder.append(ignored); + builder.append(" ignored"); + } + builder.append(")"); + return builder.toString(); + } + + @Override + protected void okCallback() { + close(); + } + + @Override + public boolean isModal() { + return false; + } + + private JPanel createWorkPanel(List results) { + JPanel panel = new JPanel(new BorderLayout()); + panel.setPreferredSize(new Dimension(1200, 400)); + BSimApplyResultsTableModel model = + new BSimApplyResultsTableModel("results model", serviceProvider, program, null, + results); + + table = new GhidraFilterTable<>(model); + table.installNavigation(serviceProvider); + + panel.add(table, BorderLayout.CENTER); + + return panel; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResultsTableModel.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResultsTableModel.java new file mode 100755 index 0000000000..863f7030e6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimApplyResultsTableModel.java @@ -0,0 +1,234 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.util.List; + +import docking.widgets.table.AbstractDynamicTableColumn; +import docking.widgets.table.TableColumnDescriptor; +import ghidra.docking.settings.Settings; +import ghidra.features.bsim.gui.search.results.apply.AbstractBSimApplyTask; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Program; +import ghidra.program.util.ProgramLocation; +import ghidra.util.datastruct.Accumulator; +import ghidra.util.exception.CancelledException; +import ghidra.util.table.AddressBasedTableModel; +import ghidra.util.table.column.GColumnRenderer; +import ghidra.util.table.field.AbstractProgramBasedDynamicTableColumn; +import ghidra.util.table.field.AbstractProgramLocationTableColumn; +import ghidra.util.task.TaskMonitor; + +/** + * This is the model that backs the table in the {@link BSimApplyResultsDisplayDialog}. It defines + * four columns for the following: + * function address being changed + * original function name + * new function name + * error/warning information. + * + * Also note that this table is address-based and will emit a GoTo service event when a row is double-clicked. + * + * @see BSimApplyResultsDisplayDialog + * @see AbstractBSimApplyTask + * + */ +public class BSimApplyResultsTableModel extends AddressBasedTableModel { + + // List of all results to be displayed in the table. + private List results; + + // Columns in the table and their positions. + static final int ADDRESS_INDEX = 0; + static final int ORIGINAL_NAME_INDEX = 1; + static final int DESTINATION_NAME_INDEX = 2; + static final int STATUS_INDEX = 3; + + public BSimApplyResultsTableModel(String title, ServiceProvider serviceProvider, + Program program, TaskMonitor monitor, List results) { + super("Rename Results", serviceProvider, program, null); + this.results = results; + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = new TableColumnDescriptor<>(); + + descriptor.addVisibleColumn(new StatusColumn()); + descriptor.addVisibleColumn(new AddressColumn()); + descriptor.addVisibleColumn(new OriginalNameColumn()); + descriptor.addVisibleColumn(new DestinationNameColumn()); + descriptor.addVisibleColumn(new MessageColumn()); + + return descriptor; + } + + @Override + public ProgramLocation getProgramLocation(int row, int column) { + Address addr = getAddress(row); + return new ProgramLocation(program, addr); + } + + @Override + protected void doLoad(Accumulator accumulator, TaskMonitor monitor) + throws CancelledException { + + for (BSimApplyResult result : results) { + accumulator.add(result); + } + } + + /** + * Returns the address for the given row. + */ + @Override + public Address getAddress(int row) { + String addressStr = (String) getValueAt(row, ADDRESS_INDEX); + return program.getAddressFactory().getAddress(addressStr); + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + private static class StatusColumn + extends AbstractProgramBasedDynamicTableColumn { + + private BSimStatusRenderer statusRenderer = new BSimStatusRenderer(); + + @Override + public String getColumnName() { + return "Status"; + } + + @Override + public BSimResultStatus getValue(BSimApplyResult rowObject, Settings settings, + Program program, ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getStatus(); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + + @Override + public GColumnRenderer getColumnRenderer() { + return statusRenderer; + } + } + + private class AddressColumn + extends AbstractProgramLocationTableColumn { + + @Override + public String getColumnName() { + return "Address"; + } + + @Override + public String getValue(BSimApplyResult rowObject, Settings settings, Program p, + ServiceProvider svcProvider) throws IllegalArgumentException { + return rowObject.getAddress().toString(); + } + + @Override + public ProgramLocation getProgramLocation(BSimApplyResult rowObject, Settings settings, + Program p, ServiceProvider svcProvider) { + return new ProgramLocation(p, rowObject.getAddress()); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + } + + /** + * Defines the column in the table for displaying the original function name (the name + * to be changed). + */ + private class OriginalNameColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Original Name"; + } + + @Override + public String getValue(BSimApplyResult rowObject, Settings settings, Object data, + ServiceProvider sp) throws IllegalArgumentException { + return rowObject.getTargetFunctionName(); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + } + + /** + * Defines the column in the table for displaying the destination function name (the + * name to use as the replacement). + * + */ + private class DestinationNameColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Name From Database"; + } + + @Override + public String getValue(BSimApplyResult rowObject, Settings settings, Object data, + ServiceProvider sp) throws IllegalArgumentException { + return rowObject.getSourceFunctionName(); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + } + + /** + * Defines the column for displaying any status information related to the rename. This + * is where error information will be displayed for rename operations that fail. + * + */ + private class MessageColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Errors/Warnings"; + } + + @Override + public String getValue(BSimApplyResult rowObject, Settings settings, Object data, + ServiceProvider sp) throws IllegalArgumentException { + return rowObject.getMessage(); + } + + @Override + public int getColumnPreferredWidth() { + return 900; + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimExecutablesSummaryModel.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimExecutablesSummaryModel.java new file mode 100755 index 0000000000..e659924b06 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimExecutablesSummaryModel.java @@ -0,0 +1,298 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.util.*; + +import docking.widgets.table.TableColumnDescriptor; +import ghidra.docking.settings.Settings; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.framework.plugintool.PluginTool; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.listing.Program; +import ghidra.program.util.ProgramLocation; +import ghidra.program.util.ProgramSelection; +import ghidra.util.datastruct.Accumulator; +import ghidra.util.exception.CancelledException; +import ghidra.util.table.GhidraProgramTableModel; +import ghidra.util.table.field.AbstractProgramBasedDynamicTableColumn; +import ghidra.util.task.TaskMonitor; + +/** + * Table model built by aggregating or "summing" the columns of rows from the QueryResultModel. + * QueryResultModel rows represent functions contained in specific executables. + * This model groups function rows from the same executable and produces a single + * row for that executable. Columns are populated roughly: + * CountColumn is the number of functions in the group + * SignificanceColumn is the sum of the individual function significances in the group + * All the other columns are inherited from properties of the single executable used + * to define the group of functions. + * ExecutableNameMatch name of the executable + * ExecutableCategoryMatch a category associated with the executable + * ExecutableDateMatch date associated with the executable + * ArchitectureMatch architecture + * CompilerMatch compiler + * RepoColumn repository containing the executable + * + */ +public class BSimExecutablesSummaryModel extends GhidraProgramTableModel { + private Collection results; + + BSimExecutablesSummaryModel(PluginTool tool, DatabaseInformation info) { + super("Executable Sum", tool, null, null); + addCustomColumns(info); + } + + private void addCustomColumns(DatabaseInformation info) { + if (info == null) { + return; // Info can be null, even if FunctionDatabase return Ready (not created yet) + } + if (info.execats != null) { + for (int i = 0; i < info.execats.size(); ++i) { + addTableColumn(new ExecutableCategoryMatchColumn(info.execats.get(i))); + } + } + if (info.dateColumnName != null) { + addTableColumn(new ExecutableDateMatchColumn(info.dateColumnName)); + } + else { + addTableColumn(new ExecutableDateMatchColumn("Ingest Date")); + } + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = + new TableColumnDescriptor(); + + descriptor.addVisibleColumn(new ExecutableNameMatchColumn()); + descriptor.addHiddenColumn(new ArchitectureMatchColumn()); + descriptor.addHiddenColumn(new CompilerMatchColumn()); + descriptor.addVisibleColumn(new CountColumn()); + descriptor.addVisibleColumn(new SignificanceColumn()); + descriptor.addHiddenColumn(new RepoColumn()); + + return descriptor; + } + + @Override + protected void doLoad(Accumulator accumulator, TaskMonitor monitor) + throws CancelledException { + if ((results == null) || (results.isEmpty())) { + return; + } + + Iterator iter = results.iterator(); + while (iter.hasNext()) { + accumulator.add(iter.next()); + } + } + + @Override + public ProgramLocation getProgramLocation(int row, int column) { + return null; + } + + @Override + public ProgramSelection getProgramSelection(int[] rows) { + // We could conceivably collect addresses of all functions that are matching into a specific executable + return new ProgramSelection(); + } + + void reload(Program newProgram, Collection set) { + setProgram(newProgram); + results = set; + reload(); + } + + void clear() { + clearData(); + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + + /** + * Column holding the name of an executable containing 1 or more functions in the result set + * + */ + private static class ExecutableNameMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Exe Name"; + } + + @Override + public String getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getExecutableRecord().getNameExec(); + } + } + + /** + * Column holding the value of an executable category for an + * executable containing 1 or more functions in the result set + * + */ + private static class ExecutableCategoryMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + private String columnName; + + public ExecutableCategoryMatchColumn(String name) { + super("ExecutableCategoryMatchColumn: " + name); + columnName = name; + } + + @Override + public String getColumnName() { + return columnName; + } + + @Override + public String getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getExecutableRecord().getExeCategoryAlphabetic(columnName); + } + + } + + /** + * Column holding the date for an executable containing 1 or more functions in the result set + * + */ + private static class ExecutableDateMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + private String columnName; + + public ExecutableDateMatchColumn(String name) { + columnName = name; + } + + @Override + public String getColumnName() { + return columnName; + } + + @Override + public Date getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getExecutableRecord().getDate(); + } + } + + /** + * Column holding the architecture for an executable containing 1 or more functions in the result set + * + */ + private static class ArchitectureMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Architecture"; + } + + @Override + public String getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getExecutableRecord().getArchitecture(); + } + } + + /** + * Column holding the compiler for an executable containing 1 or more functions in the result set + * + */ + private static class CompilerMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Compiler"; + } + + @Override + public String getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getExecutableRecord().getNameCompiler(); + } + } + + /** + * Column holding the number of functions in the result set from a single executable + * + */ + private static class CountColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Function Count"; + } + + @Override + public Integer getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return Integer.valueOf(rowObject.getFunctionCount()); + } + } + + /** + * Column holding the sum of significance scores for functions in the result set from a single executable + * + */ + private static class SignificanceColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Confidence"; + } + + @Override + public Double getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return Double.valueOf(rowObject.getSignificanceSum()); + } + } + + /** + * Column holding the repository URL for an executable containing 1 or more functions in the result set + * + */ + private static class RepoColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "URL"; + } + + @Override + public String getValue(ExecutableResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + String urlstring = rowObject.getExecutableRecord().getURLString(); + if (urlstring == null) { + return "none"; + } + return urlstring; + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimMatchResult.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimMatchResult.java new file mode 100755 index 0000000000..108b099f1c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimMatchResult.java @@ -0,0 +1,269 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import static ghidra.features.bsim.gui.search.results.BSimResultStatus.*; + +import java.util.*; + +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.features.bsim.query.protocol.*; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Program; + +/** + * A possible BSim function match. The similarity + * of this function is scored and denoted by {@link #getSimilarity() similarity}. The + * significance of the match is denoted by {@link #getSignificance()}. + */ +public class BSimMatchResult { + + private final FunctionDescription qfunc; // Original queried function + private final SimilarityNote note; + private final FunctionDescription matchfunc; + private final ExecutableRecord matchexe; + private Address originalAddress; + private int hashCode; + private BSimResultStatus status = BSimResultStatus.NOT_APPLIED; + + public BSimMatchResult(FunctionDescription queriedFunction, Address addr, + SimilarityNote similarityNote) { + qfunc = queriedFunction; + note = similarityNote; + matchfunc = note.getFunctionDescription(); + matchexe = matchfunc.getExecutableRecord(); + originalAddress = addr; + hashCode = 0; + } + + public boolean isFlagSet(int mask) { + return (matchfunc.getFlags() & mask) != 0; + } + + public FunctionDescription getOriginalFunctionDescription() { + return qfunc; + } + + public FunctionDescription getMatchFunctionDescription() { + return matchfunc; + } + + public Address getAddress() { + return originalAddress; + } + + public String getExecutableURLString() { + return matchexe.getURLString(); + } + + public BSimResultStatus getStatus() { + return status; + } + + public void setStatus(BSimResultStatus status) { + if (status == IGNORED) { + if (this.status == NAME_APPLIED || this.status == SIGNATURE_APPLIED) { + return; + } + } + this.status = status; + } + + /** + * The name of the input function to which this function is similar. + * + * @return name of the input function to which this function is similar. + */ + public String getOriginalFunctionName() { + return qfunc.getFunctionName(); + } + + public long getOriginalFunctionAddress() { + return qfunc.getAddress(); + } + + /** + * The name of the executable containing this function. + * + * @return the name of the executable containing this function. + */ + public String getExecutableName() { + return matchexe.getNameExec(); + } + + public String getExeCategoryAlphabetic(String type) { + return matchexe.getExeCategoryAlphabetic(type); + } + + public String getArchitecture() { + return matchexe.getArchitecture(); + } + + public String getCompilerName() { + return matchexe.getNameCompiler(); + } + + public String getMd5() { + return matchexe.getMd5(); + } + + /** + * The name of this function. + * + * @return the name of this function. + */ + public String getSimilarFunctionName() { + return matchfunc.getFunctionName(); + } + + public long getSimilarFunctionAddress() { + return matchfunc.getAddress(); + } + + /** + * The similarity of this function to the input function. This is a value from 0.0 to 1.0. + * + * @return the similarity of this function to the input function. + */ + public double getSimilarity() { + return note.getSimilarity(); + } + + /** + * The significance of the similarity of this function to the input function. This is a value + * that starts at 0.0, with no upper bound. Functions small in size will have a low + * significance score, as there is a chance that many small functions will have a + * similar makeup. + * + * @return the significance of the similarity of this function to the input function. + */ + public double getSignificance() { + return note.getSignificance(); + } + + public Date getDate() { + return matchexe.getDate(); + } + + @Override + public int hashCode() { + if (hashCode != 0) { + return hashCode; + } + + String executableMd5 = getMd5(); + String originalFunctionName = getOriginalFunctionName(); + long origAddr = getOriginalFunctionAddress(); + String architecture = getArchitecture(); + String similarFunctionName = getSimilarFunctionName(); + long similarAddress = getSimilarFunctionAddress(); + int prime = 31; + int result = 1; + result = prime * result + executableMd5.hashCode(); + result = prime * result + originalFunctionName.hashCode(); + result = prime * result + (int) origAddr; + result = prime * result + architecture.hashCode(); + result = prime * result + similarFunctionName.hashCode(); + result = prime * result + (int) similarAddress; + result = prime * result + (int) (origAddr >> 32); + result = prime * result + (int) (similarAddress >> 32); + hashCode = result; + return hashCode; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + + if (obj == null) { + return false; + } + + if (getClass() != obj.getClass()) { + return false; + } + + BSimMatchResult other = (BSimMatchResult) obj; + if (!getMd5().equals(other.getMd5())) { + return false; + } + + if (!getOriginalFunctionName().equals(other.getOriginalFunctionName())) { + return false; + } + + if (getOriginalFunctionAddress() != other.getOriginalFunctionAddress()) { + return false; + } + + if (!getArchitecture().equals(other.getArchitecture())) { + return false; + } + + if (!getSimilarFunctionName().equals(other.getSimilarFunctionName())) { + return false; + } + + if (getSimilarFunctionAddress() != other.getSimilarFunctionAddress()) { + return false; + } + + return true; + } + + @Override + public String toString() { + // @formatter:off + return getClass().getSimpleName() + getSimilarFunctionName() + + "\n\texecutable: " + getExecutableName() + + "\n\tsimilarity: " + getSimilarity() + + "\n\tsignificance: " + getSignificance() + + "\n\toriginal function: " + getOriginalFunctionName(); + // @formatter:on + } + + public static List generate(List results, + Program prog) { + List resultrows = new ArrayList(); + for (SimilarityResult result : results) { + FunctionDescription queriedFunction = result.getBase(); + Address origAddr = BSimMatchResultsModel.recoverAddress(queriedFunction, prog); + for (SimilarityNote note : result) { + BSimMatchResult similarFunction = + new BSimMatchResult(queriedFunction, origAddr, note); + resultrows.add(similarFunction); + } + } + return resultrows; + } + + public static List filterMatchRows(BSimFilter filter, + List rows) { + if (filter == null || filter.isEmpty()) { + return rows; + } + List filteredrows = new ArrayList(); + for (BSimMatchResult row : rows) { + if (filter.evaluate(row.getMatchFunctionDescription())) { + filteredrows.add(row); + } + } + return filteredrows; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimMatchResultsModel.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimMatchResultsModel.java new file mode 100755 index 0000000000..6d66a473b0 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimMatchResultsModel.java @@ -0,0 +1,739 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import static ghidra.features.bsim.gui.search.results.BSimResultStatus.*; + +import java.awt.Component; +import java.util.*; + +import javax.swing.JLabel; +import javax.swing.JTable; +import javax.swing.table.TableModel; + +import docking.widgets.table.*; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.util.NamespaceUtils; +import ghidra.docking.settings.Settings; +import ghidra.docking.settings.SettingsDefinition; +import ghidra.features.bsim.gui.filters.FunctionTagBSimFilterType; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.framework.plugintool.PluginTool; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.program.model.symbol.Namespace; +import ghidra.util.datastruct.Accumulator; +import ghidra.util.exception.CancelledException; +import ghidra.util.table.AddressBasedTableModel; +import ghidra.util.table.column.AbstractGColumnRenderer; +import ghidra.util.table.column.GColumnRenderer; +import ghidra.util.table.field.*; +import ghidra.util.task.TaskMonitor; + +/** + * Table model for BSim Similar function search results + */ +public class BSimMatchResultsModel extends AddressBasedTableModel { + private static final ShowNamespaceSettingsDefinition SHOW_NAMESPACE = + ShowNamespaceSettingsDefinition.DEF; + private static SettingsDefinition[] SETTINGS_DEFS = { SHOW_NAMESPACE }; + private Collection results = new ArrayList(); + + // Maps functions (represented by addresses) to the number of matches in the query. + // This is here to provide quick access for the MatchCountTableColumn. + private Map functionMatchMap = new HashMap<>(); + + public BSimMatchResultsModel(PluginTool tool, DatabaseInformation info, + LSHVectorFactory lshVectorFactory) { + super("Query Results", tool, null, null); + addCustomColumns(info, lshVectorFactory); + } + + private void addCustomColumns(DatabaseInformation info, LSHVectorFactory lshVectorFactory) { + if (info == null) { + return; // Info can be null, even if FunctionDatabase return Ready (not created yet) + } + if (info.execats != null) { + for (int i = 0; i < info.execats.size(); ++i) { + addTableColumn(new ExecCategoryColumn(info.execats.get(i))); + } + } + if (info.functionTags != null) { + int mask = 1; + mask <<= FunctionTagBSimFilterType.RESERVED_BITS; + for (int i = 0; i < info.functionTags.size(); ++i) { + addTableColumn(new FunctionTagColumn(info.functionTags.get(i), mask)); + mask <<= 1; + } + } + if (info.dateColumnName != null) { + addTableColumn(new ExecDateColumn(info.dateColumnName)); + } + else { + addTableColumn(new ExecDateColumn("Ingest Date")); + } + + // Must add this column here because it requires that the queryManager + // be available. At the time createTableColumnDescriptor() is called this + // is not the case. The index is set to '-1' so it will be placed at the end + // of the list. + if (lshVectorFactory != null) { + addTableColumn(new SelfSignificanceColumn(lshVectorFactory), -1, false); + } + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = + new TableColumnDescriptor(); + + descriptor.addVisibleColumn(new StatusColumn()); + descriptor.addVisibleColumn(new SimilarityColumn()); + descriptor.addVisibleColumn(new SignificanceColumn(), 1, false); + descriptor.addVisibleColumn(new QueryFunctionColumn()); + descriptor.addVisibleColumn(new FuncNameMatchColumn()); + descriptor.addVisibleColumn(new ExecNameMatchColumn()); + descriptor.addHiddenColumn(new ArchitectureMatchColumn()); + descriptor.addHiddenColumn(new ExecMd5Column()); + descriptor.addHiddenColumn(new CompilerMatchColumn()); + descriptor.addHiddenColumn(new MatchCountTableColumn()); + descriptor.addHiddenColumn(new FunctionSizeTableColumn()); + descriptor.addHiddenColumn(new FunctionTagColumn("Known Library", + FunctionTagBSimFilterType.KNOWN_LIBRARY_MASK)); + descriptor.addHiddenColumn(new FunctionTagColumn("Has Unimplemented", + FunctionTagBSimFilterType.HAS_UNIMPLEMENTED_MASK)); + descriptor.addHiddenColumn(new FunctionTagColumn("Has Bad Data", + FunctionTagBSimFilterType.HAS_BADDATA_MASK)); + descriptor.addVisibleColumn( + DiscoverableTableUtils.adaptColumForModel(this, new AddressTableColumn())); + descriptor.addHiddenColumn(new MatchingFunctionAddressTableColumn()); + + return descriptor; + } + + @Override + public Address getAddress(int row) { + int index = getColumnIndex(AddressTableColumn.class); + return ((AddressBasedLocation) getValueAt(row, index)).getAddress(); + } + + @Override + protected void doLoad(Accumulator accumulator, TaskMonitor monitor) + throws CancelledException { + + if (results.isEmpty()) { + return; + } + + for (BSimMatchResult similarFunction : results) { + accumulator.add(similarFunction); + } + } + + void addResult(Collection result) { + if (result == null) { + return; // not sure if this can happen + } + + for (BSimMatchResult function : result) { + addObject(function); + } + } + + void reload(Program newProgram, List rowset) { + setProgram(newProgram); + if (rowset == null) { + clear(); + return; + } + + results = rowset; + + parseFunctionMatchCounts(results); + + super.reload(); + } + + /** + * Parses the given result set to find the number of matches associated with + * each base function. + * + * @param queryResults the query results to inspect + */ + private void parseFunctionMatchCounts(Collection queryResults) { + // Begin with a clear map. + functionMatchMap.clear(); + for (BSimMatchResult result : queryResults) { + Address key = result.getAddress(); + functionMatchMap.put(key, functionMatchMap.getOrDefault(key, 0) + 1); + } + } + + void clear() { + clearData(); + } + + /** + * Associate a given FunctionDescription with the entry point of the matching function in a program + * @param desc is the FunctionDescription to recover + * @param prog is the Program (possibly) containing the Function object + * @return the entry point address of the function (if it exists), or just the address within the default space + */ + public static Address recoverAddress(FunctionDescription desc, Program prog) { + Address address = + prog.getAddressFactory().getDefaultAddressSpace().getAddress(desc.getAddress()); + // Verify that we got the right function + Function func = prog.getFunctionManager().getFunctionAt(address); + if (func != null) { + if (func.getName(true).equals(desc.getFunctionName())) { + return address; + } + } + + Function f = getUniqueFunction(desc, prog); + if (f != null) { + return f.getEntryPoint(); + } + return address; + } + + private static Function getUniqueFunction(FunctionDescription desc, Program prog) { + Function f = null; + for (Namespace namespace : NamespaceUtils.getNamespacesByName(prog, null, + desc.getFunctionName())) { + if (namespace instanceof Function) { + if (f != null) { + return null; + } + f = (Function) namespace; + } + } + return f; + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + private static class StatusColumn + extends AbstractProgramBasedDynamicTableColumn { + private BSimStatusRenderer statusRenderer = new BSimStatusRenderer(); + + @Override + public String getColumnName() { + return "Status"; + } + + @Override + public BSimResultStatus getValue(BSimMatchResult rowObject, Settings settings, + Program program, ServiceProvider serviceProvider) throws IllegalArgumentException { + BSimResultStatus status = rowObject.getStatus(); + + Function function = program.getFunctionManager().getFunctionAt(rowObject.getAddress()); + if (function == null) { + return BSimResultStatus.NO_FUNCTION; + } + + boolean nameMatches = hasMatchingFunctionName(function, rowObject); + if (status == NAME_APPLIED) { + return nameMatches ? NAME_APPLIED : APPLIED_NO_LONGER_MATCHES; + } + if (status == SIGNATURE_APPLIED) { + return nameMatches ? SIGNATURE_APPLIED : APPLIED_NO_LONGER_MATCHES; + } + if (nameMatches) { + return MATCHES; + } + return status == ERROR ? ERROR : NOT_APPLIED; + } + + private boolean hasMatchingFunctionName(Function function, BSimMatchResult result) { + String name = function.getName(true); + String matchName = result.getSimilarFunctionName(); + return name.equals(matchName); + } + + @Override + public int getColumnPreferredWidth() { + return 60; + } + + @Override + public GColumnRenderer getColumnRenderer() { + return statusRenderer; + } + } + + private static class QueryFunctionColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Function Name"; + } + + @Override + public String getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + Address address = rowObject.getAddress(); + Function function = program.getFunctionManager().getFunctionAt(address); + boolean showNamespace = SHOW_NAMESPACE.getValue(settings); + if (function != null) { + return function.getName(showNamespace); + } + return "Function Missing!"; + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + + @Override + public SettingsDefinition[] getSettingsDefinitions() { + return SETTINGS_DEFS; + } + } + + private static class FuncNameMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Matching Function Name"; + } + + @Override + public String getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + String name = rowObject.getSimilarFunctionName(); + boolean showNamespace = SHOW_NAMESPACE.getValue(settings); + if (!showNamespace) { + int lastIndexOf = name.lastIndexOf("::"); + if (lastIndexOf > 0) { + name = name.substring(lastIndexOf + 2); + } + } + return name; + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + + @Override + public SettingsDefinition[] getSettingsDefinitions() { + return SETTINGS_DEFS; + } + } + + private static class ExecNameMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Exe Name"; + } + + @Override + public String getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getExecutableName(); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + } + + /** + * Column for showing the number of matches each base function has. + * + * Note the use of the {@link BSimMatchResultsModel#functionMatchMap}; this is + * for performance reasons. We don't want this class looping over the entire + * result set calculating match counts every time the table is refreshed. + * + */ + private class MatchCountTableColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Matches"; + } + + @Override + public Integer getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider provider) throws IllegalArgumentException { + return functionMatchMap.get(rowObject.getAddress()); + } + + @Override + public int getColumnPreferredWidth() { + return 50; + } + } + + /** + * Column for showing the address of the matching function. + */ + private static class MatchingFunctionAddressTableColumn + extends AbstractProgramBasedDynamicTableColumn { + private AddressOffsetHexRenderer renderer = new AddressOffsetHexRenderer(); + + @Override + public String getColumnName() { + return "Matching Function Address"; + } + + @Override + public Long getValue(BSimMatchResult rowObject, Settings settings, Program data, + ServiceProvider serviceProvider) throws IllegalArgumentException { + Long addr = rowObject.getMatchFunctionDescription().getAddress(); + return addr; + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + + @Override + public GColumnRenderer getColumnRenderer() { + return renderer; + } + } + + private static class AddressOffsetHexRenderer extends AbstractGColumnRenderer { + + @Override + public Component getTableCellRendererComponent(GTableCellRenderingData data) { + + JLabel label = (JLabel) super.getTableCellRendererComponent(data); + label.setHorizontalAlignment(RIGHT); + Long value = (Long) data.getValue(); + + if (value != null) { + label.setText(getValueString(value)); + } + return label; + } + + @Override + protected void configureFont(JTable table, TableModel model, int column) { + setFont(fixedWidthFont); + } + + @Override + public String getFilterString(Long t, Settings settings) { + return getValueString(t); + } + + private String getValueString(Long v) { + if (v == null) { + return ""; + } + String format = ((v & 0xffffffff00000000L) == 0L) ? "%08X" : "%016X"; + return String.format(format, v); + } + } + + private static class FunctionSizeTableColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Size"; + } + + @Override + public Long getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider provider) throws IllegalArgumentException { + Address address = rowObject.getAddress(); + Function function = program.getFunctionManager().getFunctionAt(address); + return function.getBody().getNumAddresses(); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + } + + private static class ExecDateColumn + extends AbstractProgramBasedDynamicTableColumn { + private String columnName; + + ExecDateColumn(String name) { + super(); + columnName = name; + } + + @Override + public String getColumnName() { + return columnName; + } + + @Override + public Date getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getDate(); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + + } + + private static class ExecCategoryColumn + extends AbstractProgramBasedDynamicTableColumn { + private String columnName; + + ExecCategoryColumn(String name) { + super("ExecCategoryColumn: " + name); + columnName = name; + } + + @Override + public String getColumnName() { + return columnName; + } + + @Override + public String getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getExeCategoryAlphabetic(columnName); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + + } + + private static class ArchitectureMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Architecture"; + } + + @Override + public String getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getArchitecture(); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + + } + + private static class CompilerMatchColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Compiler"; + } + + @Override + public String getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getCompilerName(); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + + } + + private static class ExecMd5Column + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Md5"; + } + + @Override + public String getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getMd5(); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + + } + + private static class SimilarityColumn + extends AbstractProgramBasedDynamicTableColumn { + private DoubleRenderer doubleRenderer = new DoubleRenderer(); + + @Override + public String getColumnName() { + return "Similarity"; + } + + @Override + public Double getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getSimilarity(); + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + + @Override + public GColumnRenderer getColumnRenderer() { + return doubleRenderer; + } + } + + private static class SignificanceColumn + extends AbstractProgramBasedDynamicTableColumn { + private DoubleRenderer doubleRenderer = new DoubleRenderer(); + + @Override + public String getColumnName() { + return "Confidence"; + } + + @Override + public Double getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.getSignificance(); + } + + @Override + public GColumnRenderer getColumnRenderer() { + return doubleRenderer; + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + } + + private static class SelfSignificanceColumn + extends AbstractProgramBasedDynamicTableColumn { + + private DoubleRenderer doubleRenderer = new DoubleRenderer(); + private LSHVectorFactory vectorFactory; + + public SelfSignificanceColumn(LSHVectorFactory vectorFactory) { + this.vectorFactory = vectorFactory; + } + + @Override + public String getColumnName() { + return "Matching Function Self Significance"; + } + + @Override + public Double getValue(BSimMatchResult rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return vectorFactory.getSelfSignificance( + rowObject.getMatchFunctionDescription().getSignatureRecord().getLSHVector()); + } + + @Override + public GColumnRenderer getColumnRenderer() { + return doubleRenderer; + } + + @Override + public int getColumnPreferredWidth() { + return 100; + } + } + + private static class FunctionTagColumn + extends AbstractProgramBasedDynamicTableColumn { + private String columnName; + private int mask; // Mask for this particular boolean value + + FunctionTagColumn(String name, int m) { + super("Function Tag: " + name); + columnName = name; + mask = m; + } + + @Override + public String getColumnName() { + return columnName; + } + + @Override + public Boolean getValue(BSimMatchResult rowObject, Settings settings, Program data, + ServiceProvider serviceProvider) throws IllegalArgumentException { + return rowObject.isFlagSet(mask); + } + + @Override + public int getColumnPreferredWidth() { + return 200; + } + } + + private static class DoubleRenderer extends AbstractGColumnRenderer { + + @Override + public Component getTableCellRendererComponent(GTableCellRenderingData data) { + + JLabel label = (JLabel) super.getTableCellRendererComponent(data); + Double value = (Double) data.getValue(); + + if (value != null) { + label.setText(formatNumber(value, data.getColumnSettings())); + label.setToolTipText(value.toString()); + } + else { + label.setText(""); + label.setToolTipText(null); + } + return label; + } + + @Override + public String getFilterString(Double t, Settings settings) { + return formatNumber(t, settings); + } + + @Override + public ColumnConstraintFilterMode getColumnConstraintFilterMode() { + return ColumnConstraintFilterMode.ALLOW_CONSTRAINTS_FILTER_ONLY; + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimResultRowObjectToAddressTableRowMapper.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimResultRowObjectToAddressTableRowMapper.java new file mode 100644 index 0000000000..e8172e2bee --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimResultRowObjectToAddressTableRowMapper.java @@ -0,0 +1,35 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package ghidra.features.bsim.gui.search.results; + +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Program; +import ghidra.util.table.ProgramLocationTableRowMapper; + +/** + * Maps BSimMatchResult objects to Address to get addition table columns + */ +public class BSimResultRowObjectToAddressTableRowMapper + extends ProgramLocationTableRowMapper { + + @Override + public Address map(BSimMatchResult rowObject, Program program, + ServiceProvider serviceProvider) { + return rowObject.getAddress(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimResultStatus.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimResultStatus.java new file mode 100644 index 0000000000..e48771d847 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimResultStatus.java @@ -0,0 +1,40 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +/** + * Enum of BSim results apply statuses for when users attempt to apply function names or signatures + */ +public enum BSimResultStatus { + NOT_APPLIED("This result has not been applied."), + NAME_APPLIED("The name and namespace have been applied."), + SIGNATURE_APPLIED("The name, namespace and signature have been applied."), + MATCHES("The name already matches."), + APPLIED_NO_LONGER_MATCHES("This result has been applied, but no longer matches!"), + ERROR("An error occurred while attempting to apply this result."), + NO_FUNCTION("There is no longer a function at the result address!"), + IGNORED("The result was not applied because it already matched."); + + private String description; + + BSimResultStatus(String description) { + this.description = description; + } + + public String getDescription() { + return description; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchInfoDisplayDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchInfoDisplayDialog.java new file mode 100644 index 0000000000..cc7b193dcd --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchInfoDisplayDialog.java @@ -0,0 +1,122 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.awt.BorderLayout; +import java.awt.Font; +import java.util.List; +import java.util.stream.Collectors; + +import javax.swing.*; + +import docking.DialogComponentProvider; +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.search.dialog.BSimFilterSet; +import ghidra.features.bsim.gui.search.dialog.BSimFilterSet.FilterEntry; +import ghidra.features.bsim.gui.search.dialog.BSimSearchSettings; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.util.HelpLocation; +import ghidra.util.layout.PairLayout; + +/** + * Dialog for displaying the search criteria used to generate a BSim Similar Functions Search. + */ +public class BSimSearchInfoDisplayDialog extends DialogComponentProvider { + + private BSimServerInfo server; + private BSimSearchSettings settings; + private boolean isOverview; + + public BSimSearchInfoDisplayDialog(BSimServerInfo server, BSimSearchSettings searchSettings, + boolean isOverview) { + super("BSim Search Criteria"); + this.server = server; + this.settings = searchSettings; + this.isOverview = isOverview; + addWorkPanel(buildWorkPanel()); + String anchor = isOverview ? "Overview_Search_Info_Action" : "Search_Info_Action"; + setHelpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, anchor)); + setRememberSize(false); + addOKButton(); + } + + private JComponent buildWorkPanel() { + JPanel panel = new JPanel(new BorderLayout()); + panel.add(buildSearchInfoPanel(), BorderLayout.NORTH); + return panel; + } + + private JPanel buildSearchInfoPanel() { + JPanel panel = new JPanel(new PairLayout(0, 10)); + panel.setBorder(BorderFactory.createEmptyBorder(10, 10, 10, 10)); + + JLabel dataLabel = new JLabel("Search Options:"); + dataLabel.setFont(dataLabel.getFont().deriveFont(Font.ITALIC)); + panel.add(dataLabel); + panel.add(new JLabel("")); + + panel.add(new JLabel("BSim Server:", SwingConstants.RIGHT)); + panel.add(getDisplayField(server.getDBName())); + + panel.add(new JLabel("Similarity Threshold:", SwingConstants.RIGHT)); + panel.add(getDisplayField(Double.toString(settings.getSimilarity()))); + + panel.add(new JLabel("Confidence Threshold:", SwingConstants.RIGHT)); + panel.add(getDisplayField(Double.toString(settings.getConfidence()))); + + if (!isOverview) { + panel.add(new JLabel("Max Results:", SwingConstants.RIGHT)); + panel.add(getDisplayField(Integer.toString(settings.getMaxResults()))); + addFilters(panel); + } + return panel; + } + + private void addFilters(JPanel panel) { + panel.add(new JLabel("")); + panel.add(new JLabel("")); + JLabel filterLabel = new JLabel("Filters:"); + filterLabel.setFont(filterLabel.getFont().deriveFont(Font.ITALIC)); + panel.add(filterLabel); + panel.add(new JLabel("")); + + BSimFilterSet bSimFilterSet = settings.getBSimFilterSet(); + List filters = bSimFilterSet.getFilterEntries(); + if (filters.isEmpty()) { + panel.add(new JLabel("None", SwingConstants.RIGHT)); + return; + } + for (FilterEntry filter : filters) { + panel.add(new JLabel(filter.filterType().getLabel() + ":", SwingConstants.RIGHT)); + panel.add(getDisplayField(getValueString(filter.values()))); + } + } + + private JComponent getDisplayField(String data) { + JTextField textField = new JTextField(data); + textField.setEditable(false); + return textField; + } + + private String getValueString(List values) { + return values.stream().collect(Collectors.joining(", ")); + } + + @Override + protected void okCallback() { + close(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsFilterDialog.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsFilterDialog.java new file mode 100644 index 0000000000..611771c3bd --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsFilterDialog.java @@ -0,0 +1,90 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.awt.Dimension; +import java.util.List; + +import javax.swing.BorderFactory; +import javax.swing.JComponent; + +import docking.DialogComponentProvider; +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.gui.search.dialog.BSimFilterPanel; +import ghidra.features.bsim.gui.search.dialog.BSimFilterSet; +import ghidra.util.HelpLocation; +import ghidra.util.MessageType; + +/** + * Dialog for configuring post BSim search filters + */ +public class BSimSearchResultsFilterDialog extends DialogComponentProvider { + + private BSimFilterPanel filterPanel; + private boolean cancelled = false; + + protected BSimSearchResultsFilterDialog(List filters, BSimFilterSet filterSet) { + super("BSim Results Filters"); + setTransient(isTransient()); + + addWorkPanel(buildMainPanel(filters, filterSet)); + addOKButton(); + addCancelButton(); + setHelpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "BSim_Filters")); + } + + private JComponent buildMainPanel(List filters, BSimFilterSet filterSet) { + filterPanel = new BSimFilterPanel(filters, filterSet, this::filterValueChanged); + filterPanel.setBorder(BorderFactory.createEmptyBorder(0, 0, 0, 10)); + filterPanel.setPreferredSize(new Dimension(700, 200)); + return filterPanel; + } + + private void filterValueChanged() { + if (filterPanel.hasValidFilters()) { + setOkEnabled(true); + clearStatusText(); + } + else { + setOkEnabled(false); + setStatusText("One or more filters has invalid data!", MessageType.ERROR); + } + } + + public BSimFilterSet getFilters() { + if (cancelled) { + return null; + } + return filterPanel.getFilterSet(); + } + + @Override + protected void cancelCallback() { + cancelled = true; + close(); + } + + @Override + protected void okCallback() { + if (!filterPanel.hasValidFilters()) { + setStatusText("One or more filters is invalid!", MessageType.ERROR); + return; + } + close(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsProvider.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsProvider.java new file mode 100644 index 0000000000..f0c686051e --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsProvider.java @@ -0,0 +1,593 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.awt.*; +import java.awt.event.MouseEvent; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.*; +import java.util.List; +import java.util.function.IntSupplier; + +import javax.swing.*; + +import docking.*; +import docking.action.ToggleDockingAction; +import docking.action.builder.ActionBuilder; +import docking.action.builder.ToggleActionBuilder; +import docking.widgets.table.RowObjectTableModel; +import generic.lsh.vector.LSHVectorFactory; +import generic.theme.GIcon; +import ghidra.app.plugin.core.functioncompare.FunctionComparisonProvider; +import ghidra.app.services.*; +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.gui.filters.Md5BSimFilterType; +import ghidra.features.bsim.gui.search.dialog.*; +import ghidra.features.bsim.gui.search.results.apply.*; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.facade.SFQueryInfo; +import ghidra.features.bsim.query.facade.SFQueryResult; +import ghidra.features.bsim.query.protocol.BSimFilter; +import ghidra.framework.model.*; +import ghidra.framework.plugintool.ComponentProviderAdapter; +import ghidra.framework.plugintool.PluginTool; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.*; +import ghidra.program.util.ChangeManager; +import ghidra.util.HelpLocation; +import ghidra.util.Msg; +import ghidra.util.datastruct.Counter; +import ghidra.util.table.GhidraFilterTable; +import ghidra.util.table.SelectionNavigationAction; +import ghidra.util.task.TaskLauncher; +import resources.Icons; + +/** + * ComponentProvider for displaying BSim Similar Functions Search results. + */ +public class BSimSearchResultsProvider extends ComponentProviderAdapter { + private static final Icon FUNCTIONS_ICON = new GIcon("icon.bsim.functions.table"); + private static final String PROVIDER_WINDOW_GROUP = "bsim.results"; + private static final String NAME = "BSim Search Results"; + private static final Icon SPLIT_VIEW_ICON = new GIcon("icon.bsim.table.split"); + private static final String APPLY_GROUP = "0"; + private BSimServerInfo serverInfo; + private SFQueryInfo queryInfo; + private Program program; + private JComponent mainComponent; + private JPanel mainPanel; + private BSimMatchResultsModel matchesModel; + private BSimExecutablesSummaryModel executablesModel; + private DatabaseInformation dbInfo; + private LSHVectorFactory lshVectorFactory; + private BSimFilterSet postFilters = new BSimFilterSet(); + private List rows = new ArrayList<>(); + private GhidraFilterTable matchesTable; + + private JPanel matchesPanel; + private GhidraFilterTable executablesTable; + private JPanel executablesPanel; + private ToggleDockingAction showExecutableTableAction; + private BSimSearchPlugin plugin; + private DomainObjectListener listener = new MyDomainObjectListener(); + private BSimSearchSettings settings; + + public BSimSearchResultsProvider(BSimSearchPlugin plugin, PluginTool tool, + BSimServerInfo serverInfo, DatabaseInformation dbInfo, + LSHVectorFactory lshVectorFactory, SFQueryInfo queryInfo, BSimSearchSettings settings) { + super(tool, NAME, plugin.getName()); + this.plugin = plugin; + this.serverInfo = serverInfo; + this.dbInfo = dbInfo; + this.lshVectorFactory = lshVectorFactory; + this.queryInfo = queryInfo; + this.settings = settings; + this.program = queryInfo.getProgram(); + + setWindowGroup(PROVIDER_WINDOW_GROUP); + setDefaultWindowPosition(WindowPosition.WINDOW); + setTitle(NAME); + setSubTitle(getIdString()); + setTabText("BSim " + getIdString()); + setWindowMenuGroup("BSim"); + setHelpLocation(new HelpLocation("BSimSearchPlugin", "Similar_Functions_Results")); + setTransient(); + + mainComponent = buildComponent(); + + tool.addComponentProvider(this, true); + + createActions(); + matchesTable.installNavigation(tool.getService(GoToService.class)); + matchesTable.setNavigateOnSelectionEnabled(true); + program.addListener(listener); + } + + @Override + public void componentHidden() { + super.componentHidden(); + if (plugin != null) { + plugin.providerClosed(this); + } + } + + private void createActions() { + new ActionBuilder("Search Info", getName()).toolBarIcon(Icons.INFO_ICON) + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Search_Info_Action")) + .onAction(c -> showSearchInfo()) + .buildAndInstallLocal(this); + + new ActionBuilder("Show Searched Functions", getName()).toolBarIcon(FUNCTIONS_ICON) + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Searched_Functions")) + .onAction(c -> showSearchedFunctions()) + .buildAndInstallLocal(this); + + new ActionBuilder("Filter Results", getName()).toolBarIcon(Icons.CONFIGURE_FILTER_ICON) + .helpLocation( + new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Filter_Results_Action")) + .onAction(c -> showFilterPanel()) + .buildAndInstallLocal(this); + + new ActionBuilder("Filter Executable", getName()).popupMenuPath("Filter on this Executable") + .description("Filter on a specific executable in the function match table") + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Filter_On_Executable")) + .withContext(ExecutableTableActionContext.class) + .enabledWhen(c -> c.getSelectedRowCount() == 1) + .onAction(this::filterOnExecutable) + .buildAndInstallLocal(this); + + new ActionBuilder("Load Executable", getName()).popupMenuPath("Load Executable") + .description("Load the selected executable into the Codebrowser") + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Load_Executable")) + .withContext(ExecutableTableActionContext.class) + .enabledWhen(c -> c.getSelectedRowCount() == 1) + .onAction(this::loadExecutable) + .buildAndInstallLocal(this); + + new ActionBuilder("Compare Functions", getName()).popupMenuPath("Compare Functions") + .popupMenuGroup("1") + .keyBinding("shift c") + .sharedKeyBinding() + .description("Compares the Functions with its remote match") + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Compare_Functions")) + .withContext(BSimMatchesTableActionContext.class) + .enabledWhen(c -> c.getSelectedRowCount() > 0) + .onAction(this::compareFunctions) + .buildAndInstallLocal(this); + + new ActionBuilder("Apply Function Name", getName()).popupMenuPath("Apply Name") + .popupMenuGroup(APPLY_GROUP, "1") + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Apply_Name")) + .withContext(BSimMatchesTableActionContext.class) + .enabledWhen(c -> c.getSelectedRowCount() > 0) + .onAction(this::applyName) + .buildAndInstallLocal(this); + + new ActionBuilder("Apply Function Signature", getName()).popupMenuPath("Apply Signature") + .popupMenuGroup(APPLY_GROUP, "2") + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Apply_Signature")) + .withContext(BSimMatchesTableActionContext.class) + .enabledWhen(c -> c.getSelectedRowCount() > 0) + .onAction(this::applySignature) + .buildAndInstallLocal(this); + + new ActionBuilder("Apply Signature and Datatypes", getName()) + .popupMenuPath("Apply Signature and Data Types") + .popupMenuGroup(APPLY_GROUP, "3") + .helpLocation( + new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Apply_Signature_With_Datatypes")) + .withContext(BSimMatchesTableActionContext.class) + .enabledWhen(c -> c.getSelectedRowCount() > 0) + .onAction(this::applySignatureWithDatatypes) + .buildAndInstallLocal(this); + + showExecutableTableAction = new ToggleActionBuilder("Show Executables Table", getName()) + .toolBarIcon(SPLIT_VIEW_ICON) + .description("Toggles showing Executables table") + .helpLocation( + new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Hide_Show_Executables_Table")) + .selected(true) + .onAction(c -> showExecutableTable(showExecutableTableAction.isSelected())) + .buildAndInstallLocal(this); + + addLocalAction(new SelectionNavigationAction(plugin, matchesTable.getTable())); + + new ActionBuilder("Clear BSim Error Status", getName()).popupMenuPath("Clear error status") + .helpLocation(new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Clear_Error_Status")) + .withContext(BSimMatchesTableActionContext.class) + .enabledWhen(this::canClearErrors) + .onAction(this::clearErrors) + .buildAndInstallLocal(this); + } + + private void applyName(BSimMatchesTableActionContext c) { + List selected = matchesTable.getSelectedRowObjects(); + AbstractBSimApplyTask task = + new NameAndNamespaceBSimApplyTask(program, selected, dockingTool); + TaskLauncher.launch(task); + matchesModel.fireTableDataChanged(); + } + + private void applySignature(BSimMatchesTableActionContext c) { + List selected = matchesTable.getSelectedRowObjects(); + AbstractBSimApplyTask task = + new SignatureBSimApplyTask(program, selected, true, dockingTool); + TaskLauncher.launch(task); + matchesModel.fireTableDataChanged(); + } + + private void applySignatureWithDatatypes(BSimMatchesTableActionContext c) { + List selected = matchesTable.getSelectedRowObjects(); + AbstractBSimApplyTask task = + new SignatureBSimApplyTask(program, selected, false, dockingTool); + TaskLauncher.launch(task); + matchesModel.fireTableDataChanged(); + } + + private boolean canClearErrors(BSimMatchesTableActionContext c) { + int rowCount = c.getSelectedRowCount(); + if (rowCount == 0) { + return false; + } + if (rowCount == 1) { + BSimMatchResult selectedRowObject = matchesTable.getSelectedRowObject(); + return selectedRowObject.getStatus() == BSimResultStatus.ERROR; + } + return true; + } + + private void clearErrors(BSimMatchesTableActionContext c) { + List selected = matchesTable.getSelectedRowObjects(); + for (BSimMatchResult result : selected) { + if (result.getStatus() == BSimResultStatus.ERROR) { + result.setStatus(BSimResultStatus.NOT_APPLIED); + } + } + matchesModel.fireTableDataChanged(); + } + + private void compareFunctions(BSimMatchesTableActionContext c) { + FunctionComparisonService service = tool.getService(FunctionComparisonService.class); + if (service == null) { + Msg.error(this, "Function Comparison Service not found!"); + return; + } + FunctionComparisonProvider comparisonProvider = service.createFunctionComparisonProvider(); + comparisonProvider.removeAddFunctionsAction(); + List selectedRowObjects = matchesTable.getSelectedRowObjects(); + Set openedPrograms = new HashSet<>(); + for (BSimMatchResult row : selectedRowObjects) { + try { + Function originalFunction = getOriginalFunction(row); + Function matchFunction = getMatchFunction(row, openedPrograms); + comparisonProvider.getModel().compareFunctions(originalFunction, matchFunction); + } + catch (FunctionComparisonException e) { + Msg.showError(this, null, "Unable to Compare Functions", + "Compare Functions: " + e.getMessage()); + } + } + comparisonProvider.setCloseListener(() -> { + for (Program remote : openedPrograms) { + remote.release(BSimSearchResultsProvider.this); + } + }); + } + + private void filterOnExecutable(ExecutableTableActionContext c) { + ExecutableRecord exerecord = c.getSelectedExecutableResult().getExecutableRecord(); + Md5BSimFilterType filterType = new Md5BSimFilterType(); + postFilters.removeAll(filterType); + postFilters.addEntry(filterType, List.of(exerecord.getMd5())); + updateTableData(); + } + + private void loadExecutable(ExecutableTableActionContext c) { + ExecutableResult result = c.getSelectedExecutableResult(); + ExecutableRecord record = result.getExecutableRecord(); + String programUrl = record.getURLString(); + openRemoteProgramInTool(programUrl); + } + + private void showFilterPanel() { + List filters = BSimFilterType.generateBsimFilters(dbInfo, false); + BSimSearchResultsFilterDialog filterDialog = + new BSimSearchResultsFilterDialog(filters, postFilters); + tool.showDialog(filterDialog); + BSimFilterSet results = filterDialog.getFilters(); + if (results != null) { + postFilters = results; + updateTableData(); + } + } + + private void updateTableData() { + BSimFilter filter = postFilters.getBSimFilter(); + + List filteredrows = BSimMatchResult.filterMatchRows(filter, rows); + Set execrows = ExecutableResult.generateFromMatchRows(filteredrows); + + matchesModel.reload(program, filteredrows); + executablesModel.reload(program, execrows); + } + + private void showSearchInfo() { + tool.showDialog(new BSimSearchInfoDisplayDialog(serverInfo, settings, false)); + } + + private void showSearchedFunctions() { + GoToService service = tool.getService(GoToService.class); + HelpLocation help = new HelpLocation(BSimSearchPlugin.HELP_TOPIC, "Searched_Functions"); + + tool.showDialog(new SelectedFunctionsTableDialog(queryInfo.getFunctions(), service, help, + getFunctionMatchCount())); + } + + private Map getFunctionMatchCount() { + Map map = new HashMap<>(); + for (BSimMatchResult result : rows) { + Address address = result.getAddress(); + Counter counter = map.computeIfAbsent(address, a -> new Counter()); + counter.increment(); + } + + Map countMap = new HashMap<>(); + for (FunctionSymbol functionSymbol : queryInfo.getFunctions()) { + Counter counter = map.get(functionSymbol.getAddress()); + countMap.put(functionSymbol.getAddress(), counter == null ? 0 : counter.count()); + } + return countMap; + } + + private JComponent buildComponent() { + mainPanel = new JPanel(); + mainPanel.setLayout(new BorderLayout()); + mainPanel.setPreferredSize(new Dimension(1000, 800)); + matchesModel = new BSimMatchResultsModel(tool, dbInfo, lshVectorFactory); + matchesTable = new GhidraFilterTable<>(matchesModel); + matchesPanel = buildTitledTablePanel("Function Matches", matchesTable, () -> rows.size()); + + executablesModel = new BSimExecutablesSummaryModel(tool, dbInfo); + executablesTable = new GhidraFilterTable<>(executablesModel); + executablesPanel = buildTitledTablePanel("Executables", executablesTable, + () -> executablesModel.getUnfilteredRowCount()); + + showExecutableTable(true); + + return mainPanel; + } + + private void showExecutableTable(boolean selected) { + mainPanel.removeAll(); + if (selected) { + JSplitPane split = + new JSplitPane(JSplitPane.VERTICAL_SPLIT, matchesPanel, executablesPanel); + + split.setResizeWeight(0.5); + split.setDividerSize(10); + + mainPanel.add(split, BorderLayout.CENTER); + } + else { + mainPanel.add(matchesPanel, BorderLayout.CENTER); + } + + mainPanel.validate(); + } + + @Override + public JComponent getComponent() { + return mainComponent; + } + + private String getIdString() { + String serverShortName = serverInfo.getShortDBName(); + StringBuilder builder = new StringBuilder("[server: "); + builder.append(serverShortName); + builder.append(", function: "); + + Set functions = queryInfo.getFunctions(); + if (functions.size() == 1) { + builder.append(functions.iterator().next().getName()); + } + else { + builder.append(functions.size() + " selected"); + } + builder.append(", Similarity: "); + builder.append(queryInfo.getSimilarityThreshold()); + builder.append(", Confidence: "); + builder.append(queryInfo.getSignificanceThreshold()); + builder.append("]"); + return builder.toString(); + } + + public void setFinalQueryResults(SFQueryResult result) { + rows = BSimMatchResult.generate(result.getSimilarityResults(), program); + updateTableData(); + } + + @Override + public ActionContext getActionContext(MouseEvent event) { + if (event != null) { + return getActionContext(event.getSource()); + } + + KeyboardFocusManager kfm = KeyboardFocusManager.getCurrentKeyboardFocusManager(); + return getActionContext(kfm.getFocusOwner()); + } + + private ActionContext getActionContext(Object source) { + if (source == matchesTable.getTable()) { + return new BSimMatchesTableActionContext(); + } + else if (source == executablesTable.getTable()) { + return new ExecutableTableActionContext(); + } + return new DefaultActionContext(this); + } + + private Program openRemoteProgramInTool(String urlString) { + ProgramManager service = tool.getService(ProgramManager.class); + + try { + URL url = new URL(urlString); + return service.openProgram(url, ProgramManager.OPEN_CURRENT); + } + catch (MalformedURLException exc) { + return null; + } + } + + private Program getCachedRemoteProgram(String urlString, Set openedPrograms) { + ProgramManager service = tool.getService(ProgramManager.class); + + try { + URL url = new URL(urlString); + Program remote = service.openCachedProgram(url, this); + if (remote == null) { + return null; + } + if (!openedPrograms.add(remote)) { + // The service added 'this' as a consumer. We previously opened it and we don't + // want the program to have the same consumer twice. + remote.release(this); + } + return remote; + } + catch (MalformedURLException exc) { + return null; + } + } + + private JPanel buildTitledTablePanel(String title, GhidraFilterTable table, + IntSupplier nonFilteredRowCount) { + JPanel panel = new JPanel(new BorderLayout()); + panel.setBorder(BorderFactory.createEmptyBorder(10, 2, 10, 2)); + JLabel titleLabel = new JLabel(title); + panel.add(titleLabel, BorderLayout.NORTH); + panel.add(table, BorderLayout.CENTER); + + RowObjectTableModel model = table.getModel(); + model.addTableModelListener((e) -> { + int rowCount = model.getRowCount(); + String text = title + " - " + rowCount + " results"; + int nonFilteredSize = nonFilteredRowCount.getAsInt(); + if (nonFilteredSize != rowCount) { + text += " (Filtered from " + nonFilteredSize + " results)"; + } + titleLabel.setText(text); + }); + return panel; + } + + private Function getOriginalFunction(BSimMatchResult resultRow) + throws FunctionComparisonException { + // Determine the original function (local program's function; left side) + Address originalEntryPoint = resultRow.getAddress(); + FunctionManager originalFunctionManager = program.getFunctionManager(); + Function originalFunction = originalFunctionManager.getFunctionAt(originalEntryPoint); + if (originalFunction == null) { + throw new FunctionComparisonException("Couldn't get local function " + + resultRow.getOriginalFunctionDescription().getFunctionName() + " at " + + originalEntryPoint.toString() + "."); + } + return originalFunction; + } + + private Function getMatchFunction(BSimMatchResult resultRow, Set opened) + throws FunctionComparisonException { + // Determine the match function (remote program's function; right side) + Program matchProgram = getCachedRemoteProgram(resultRow.getExecutableURLString(), opened); + if (matchProgram == null) { + throw new FunctionComparisonException( + "Couldn't open remote program: " + resultRow.getExecutableURLString() + " for " + + resultRow.getSimilarFunctionName() + "."); + } + FunctionDescription matchFunctionDescription = resultRow.getMatchFunctionDescription(); + long matchOffset = matchFunctionDescription.getAddress(); + Address matchEntryPoint = + matchProgram.getAddressFactory().getDefaultAddressSpace().getAddress(matchOffset); + FunctionManager matchFunctionManager = matchProgram.getFunctionManager(); + Function matchFunction = matchFunctionManager.getFunctionAt(matchEntryPoint); + if (matchFunction == null) { + throw new FunctionComparisonException("Couldn't get remote function " + + matchFunctionDescription.getFunctionName() + " at " + matchEntryPoint + "."); + } + return matchFunction; + } + + private class BSimMatchesTableActionContext extends DefaultActionContext { + + BSimMatchesTableActionContext() { + super(BSimSearchResultsProvider.this); + } + + public int getSelectedRowCount() { + return matchesTable.getTable().getSelectedRowCount(); + } + + } + + private class ExecutableTableActionContext extends DefaultActionContext { + + ExecutableTableActionContext() { + super(BSimSearchResultsProvider.this); + } + + public ExecutableResult getSelectedExecutableResult() { + return executablesTable.getSelectedRowObject(); + } + + public int getSelectedRowCount() { + return executablesTable.getTable().getSelectedRowCount(); + } + } + + public Program getProgram() { + return program; + } + +//================================================================================================== +// Test methods +//================================================================================================== + BSimMatchResultsModel getMatchesModel() { + return matchesModel; + } + + BSimExecutablesSummaryModel getExecutablesModel() { + return executablesModel; + } + +//================================================================================================== +// Classes +//================================================================================================== + private class MyDomainObjectListener implements DomainObjectListener { + + @Override + public void domainObjectChanged(DomainObjectChangedEvent ev) { + if (ev.containsEvent(ChangeManager.DOCR_SYMBOL_RENAMED) || + ev.containsEvent(DomainObject.DO_OBJECT_RESTORED)) { + matchesModel.fireTableDataChanged(); + } + + } + + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimStatusRenderer.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimStatusRenderer.java new file mode 100644 index 0000000000..743e735fa5 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/BSimStatusRenderer.java @@ -0,0 +1,84 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.awt.Component; + +import javax.swing.Icon; +import javax.swing.JLabel; + +import docking.widgets.table.GTableCellRenderingData; +import generic.theme.GIcon; +import ghidra.docking.settings.Settings; +import ghidra.util.table.column.AbstractGColumnRenderer; +import resources.Icons; + +/** + * Renderer for display BSim apply results from attempting to apply function names and signatures + * from BSim Search results. + */ +public class BSimStatusRenderer extends AbstractGColumnRenderer { + private static final Icon NOT_APPLIED_ICON = null; + private static final Icon NAME_APPLIED_ICON = + new GIcon("icon.bsim.results.status.name.applied"); + private static final Icon SIGNATURE_APPLIED_ICON = + new GIcon("icon.bsim.results.status.signature.applied"); + private static final Icon ERROR_ICON = Icons.ERROR_ICON; + private static final Icon MATCHES_ICON = new GIcon("icon.bsim.results.status.matches"); + private static final Icon APPLIED_NO_LONGER_MATCHES_ICON = Icons.WARNING_ICON; + private static final Icon NO_FUNCTION_ICON = Icons.STRONG_WARNING_ICON; + private static final Icon IGNORED_ICON = new GIcon("icon.bsim.results.status.ignored"); + + @Override + public Component getTableCellRendererComponent(GTableCellRenderingData data) { + + JLabel label = (JLabel) super.getTableCellRendererComponent(data); + BSimResultStatus status = (BSimResultStatus) data.getValue(); + label.setText(""); + label.setIcon(getIcon(status)); + label.setToolTipText(status.getDescription()); + return label; + } + + private Icon getIcon(BSimResultStatus status) { + switch (status) { + case NAME_APPLIED: + return NAME_APPLIED_ICON; + case SIGNATURE_APPLIED: + return SIGNATURE_APPLIED_ICON; + case ERROR: + return ERROR_ICON; + case MATCHES: + return MATCHES_ICON; + case NOT_APPLIED: + return NOT_APPLIED_ICON; + case APPLIED_NO_LONGER_MATCHES: + return APPLIED_NO_LONGER_MATCHES_ICON; + case NO_FUNCTION: + return NO_FUNCTION_ICON; + case IGNORED: + return IGNORED_ICON; + default: + return ERROR_ICON; + } + } + + @Override + public String getFilterString(BSimResultStatus t, Settings settings) { + return t.toString(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/DisplayFunctionsPanel.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/DisplayFunctionsPanel.java new file mode 100755 index 0000000000..328acd1f66 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/DisplayFunctionsPanel.java @@ -0,0 +1,285 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.awt.BorderLayout; +import java.awt.Color; +import java.util.*; + +import javax.swing.JPanel; +import javax.swing.event.HyperlinkEvent; + +import docking.widgets.HyperlinkComponent; +import docking.widgets.table.DiscoverableTableUtils; +import docking.widgets.table.TableColumnDescriptor; +import generic.theme.GThemeDefaults.Colors.Palette; +import ghidra.app.plugin.core.table.TableComponentProvider; +import ghidra.app.util.query.TableService; +import ghidra.docking.settings.Settings; +import ghidra.features.bsim.query.facade.SFQueryResult; +import ghidra.features.bsim.query.protocol.SimilarityResult; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.program.util.ProgramLocation; +import ghidra.util.Msg; +import ghidra.util.datastruct.Accumulator; +import ghidra.util.exception.CancelledException; +import ghidra.util.table.AddressBasedTableModel; +import ghidra.util.table.field.*; +import ghidra.util.task.TaskMonitor; + +/** + * Panel that displays the list of functions to be included in a BSim query. + * + */ +class DisplayFunctionsPanel extends JPanel { + + private static final String SHOW_TABLE_HREF_NAME = "ShowTable"; + + private static final Color MARKER_COLOR = Palette.getColor("lightskyblue"); + + private HyperlinkComponent functionsHTMLComponent; + + private TableComponentProvider tableProvider; + private final ServiceProvider serviceProvider; + private Set selectedFunctions; + + // Maps input functions to the number of matches associated with it. This is + // here to provide quick access for the MatchCountTableColumn. + private Map functionMatchMap = new HashMap<>(); + private String description; + + DisplayFunctionsPanel(ServiceProvider serviceProvider, String desc) { + super(new BorderLayout()); + this.serviceProvider = serviceProvider; + this.description = desc; + + functionsHTMLComponent = new HyperlinkComponent(getNoFunctionsSelectedMessage()); + functionsHTMLComponent.addHyperlinkListener(SHOW_TABLE_HREF_NAME, e -> { + if (e.getEventType() != HyperlinkEvent.EventType.ACTIVATED) { + // not a mouse click + return; + } + showAllSelectedFunctionsTable(); + }); + + add(functionsHTMLComponent, BorderLayout.CENTER); + } + + /** + * Takes a new set of query results and parses the function counts. + * + * @param queryResult the object to hold the load results + */ + public void loadQueryResults(SFQueryResult queryResult) { + parseFunctionMatchCounts(queryResult); + } + + private String getNoFunctionsSelectedMessage() { + StringBuilder buffy = new StringBuilder(); + buffy.append( + "No functions selected"); + return buffy.toString(); + } + + void close() { + if (tableProvider != null) { + tableProvider.removeFromTool(); + tableProvider = null; + } + + selectedFunctions = null; + } + + void setSelectedFunctions(Set functions) { + + this.selectedFunctions = functions; + if (tableProvider != null && tableProvider.isInTool()) { + tableProvider.removeFromTool(); + tableProvider = null; + } + + if (functions.isEmpty()) { + functionsHTMLComponent.setText(getNoFunctionsSelectedMessage()); + return; + } + + String text = createTextForSelectedFunctions(functions); + functionsHTMLComponent.setText(text); + } + + private String createTextForSelectedFunctions(Set functions) { + + if (functions.isEmpty()) { + return ""; + } + StringBuilder buffy = new StringBuilder(); + int count = functions.size(); + Function firstFunc = (Function) functions.iterator().next().getObject(); + String programName = firstFunc.getProgram().getDomainFile().getPathname(); + buffy.append(description).append(" "); + buffy.append(firstFunc.getName()); + if (count > 1) { + buffy.append(" and "); + buffy.append(count - 1); + buffy.append(" other function"); + } + if (count > 2) { + buffy.append("s"); + } + buffy.append(" from "); + buffy.append(programName); + buffy.append(" "); // open anchor + buffy.append(""); + buffy.append(" (show table) "); + buffy.append(""); + buffy.append(""); // close anchor + return buffy.toString(); + } + + private void showAllSelectedFunctionsTable() { + if (tableProvider != null) { + if (tableProvider.isInTool()) { + tableProvider.setVisible(true); + return; + } + + // it has been closed--cleanup + tableProvider = null; + } + + TableService tableService = serviceProvider.getService(TableService.class); + if (tableService == null) { + Msg.showWarn(getClass(), this, "No Table Service Found", + "Unable to locate the Table Service. Make sure the plugin is installed."); + return; + } + + FunctionSymbol arbitraryFunction = selectedFunctions.iterator().next(); + Program program = arbitraryFunction.getProgram(); + SelectedFunctionsModel model = + new SelectedFunctionsModel(program, serviceProvider, selectedFunctions); + tableProvider = tableService.showTableWithMarkers("Selected Query Functions", + "QueryDialogTable", model, MARKER_COLOR, null /*icon*/, "Selected Query Functions", + null /*navigatable - use default*/); + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + + private class SelectedFunctionsModel extends AddressBasedTableModel { + private final List functions; + + SelectedFunctionsModel(Program program, ServiceProvider serviceProvider, + Set functions) { + super("Selected Query Functions", serviceProvider, program, null); + this.functions = new ArrayList<>(functions); + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = new TableColumnDescriptor<>(); + + descriptor.addVisibleColumn( + DiscoverableTableUtils.adaptColumForModel(this, new LabelTableColumn())); + descriptor.addVisibleColumn( + DiscoverableTableUtils.adaptColumForModel(this, new AddressTableColumn()), 1, true); + descriptor.addVisibleColumn(DiscoverableTableUtils.adaptColumForModel(this, + new FunctionSignatureTableColumn())); + descriptor.addVisibleColumn(new MatchCountTableColumn()); + + return descriptor; + } + + @Override + public Address getAddress(int row) { + Function function = getRowObject(row); + return function.getEntryPoint(); + } + + @Override + protected void doLoad(Accumulator accumulator, TaskMonitor monitor) + throws CancelledException { + for (FunctionSymbol sym : functions) { + Object obj = sym.getObject(); + if (obj != null) { + accumulator.add((Function) obj); + } + } + } + + @Override + public ProgramLocation getProgramLocation(int row, int column) { +// TODO: I cannot reconcile how to show the user a table of functions and let them +// navigate in a way that is understandable (if they navigate, that changes the selected +// functions, which would then clear the table) +// +// Sad Face :( +// + return null; + } + } + + /** + * Calculates how many matches are associated with each base function in + * the given result set and stores them in the {@link DisplayFunctionsPanel#functionMatchMap}. + * + * @param queryResult the object that holds the function matches + */ + private void parseFunctionMatchCounts(SFQueryResult queryResult) { + + List results = queryResult.getSimilarityResults(); + + for (SimilarityResult result : results) { + String funcName = result.getBase().getFunctionName(); + functionMatchMap.put(funcName, result.getTotalCount()); + } + } + + /** + * Column for showing the number of matches each base function has. + * + * To make this as fast as possible, the counts for each function are NOT determined + * here; they're calculated when a new result set is received and stored in the + * {@link DisplayFunctionsPanel#functionMatchMap}. This class has only to + * go to that map and extract the correct value. + * + * @see DisplayFunctionsPanel#parseFunctionMatchCounts(SFQueryResult) + * + */ + private class MatchCountTableColumn + extends AbstractProgramBasedDynamicTableColumn { + + @Override + public String getColumnName() { + return "Matches"; + } + + @Override + public Integer getValue(Function rowObject, Settings settings, Program program, + ServiceProvider serviceProvider) throws IllegalArgumentException { + + if (functionMatchMap == null) { + return 0; + } + return functionMatchMap.get(rowObject.getName()); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/ExecutableResult.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/ExecutableResult.java new file mode 100755 index 0000000000..6353d156eb --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/ExecutableResult.java @@ -0,0 +1,134 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import java.util.*; + +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.description.FunctionDescription; + +public class ExecutableResult implements Comparable { + private ExecutableRecord exerecord; + private int funccount; // Number of functions matching into this executable + private double sumsignif; // Sum of all matching function significance + private int hashCode; + + public ExecutableResult() { + exerecord = null; + funccount = 0; + sumsignif = 0.0; + } + + public ExecutableResult(ExecutableRecord rec) { + exerecord = rec; + funccount = 0; + sumsignif = 0.0; + } + + public void addFunction(double signif) { + funccount += 1; + sumsignif += signif; + } + + public ExecutableRecord getExecutableRecord() { + return exerecord; + } + + /** + * @return number of functions with matches into this executable + */ + public int getFunctionCount() { + return funccount; + } + + /** + * @return sum of significance scores for all matching functions + */ + public double getSignificanceSum() { + return sumsignif; + } + + @Override + public boolean equals(Object obj) { + if (obj == null) + return false; + if (this == obj) + return true; + ExecutableResult op2 = (ExecutableResult) obj; + return exerecord.equals(op2.exerecord); + } + + @Override + public int compareTo(ExecutableResult o) { + return exerecord.compareTo(o.exerecord); + } + + @Override + public int hashCode() { + if (hashCode != 0) { + return hashCode; + } + + hashCode = exerecord.hashCode(); + return hashCode; + } + + private static void finalizeExecutableResult(TreeSet singlefunc, + TreeSet globalfunc) { + Iterator finaliter = singlefunc.iterator(); + while (finaliter.hasNext()) { + ExecutableResult eres = finaliter.next(); + ExecutableResult tmpres = globalfunc.floor(eres); + if ((tmpres == null) || (!tmpres.equals(eres))) { + tmpres = new ExecutableResult(eres.exerecord); + globalfunc.add(tmpres); + } + tmpres.addFunction(eres.getSignificanceSum()); + } + } + + public static TreeSet generateFromMatchRows( + List filteredrows) { + TreeSet execrows = new TreeSet(); + ExecutableResult curres = new ExecutableResult(); + TreeSet exetree = new TreeSet(); + FunctionDescription curdescription = null; + Iterator iter = filteredrows.iterator(); + while (iter.hasNext()) { + BSimMatchResult simres = iter.next(); + double signif = simres.getSignificance(); + if (curdescription != simres.getOriginalFunctionDescription()) { + finalizeExecutableResult(exetree, execrows); + curdescription = simres.getOriginalFunctionDescription(); + exetree = new TreeSet(); + } + curres.exerecord = simres.getMatchFunctionDescription().getExecutableRecord(); + ExecutableResult tmpres = exetree.floor(curres); + if ((tmpres == null) || (!tmpres.equals(curres))) { // Haven't seen this executable before + tmpres = new ExecutableResult(curres.exerecord); + tmpres.sumsignif = signif; + exetree.add(tmpres); + } + else { // Seen this executable before for this particular function + if (tmpres.getSignificanceSum() < signif) + tmpres.sumsignif = signif; + } + } + if (!exetree.isEmpty()) + finalizeExecutableResult(exetree, execrows); + return execrows; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/FunctionComparisonException.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/FunctionComparisonException.java new file mode 100755 index 0000000000..df7f091c43 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/FunctionComparisonException.java @@ -0,0 +1,43 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import ghidra.util.exception.UsrException; + +/** + * An exception that can be thrown if an error is encountered while trying to compare two functions + * or apply information between them. + */ +public class FunctionComparisonException extends UsrException { + + /** + * Constructor + * @param msg a message indicating details of the error. + */ + public FunctionComparisonException(String msg) { + super(msg); + } + + /** + * Constructor + * @param msg a message indicating details of the error. + * @param cause another exception indicating the cause that led to this error exception. + */ + public FunctionComparisonException(String msg, Throwable cause) { + super(msg, cause); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/ShowNamespaceSettingsDefinition.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/ShowNamespaceSettingsDefinition.java new file mode 100644 index 0000000000..41491d0975 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/ShowNamespaceSettingsDefinition.java @@ -0,0 +1,92 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +import ghidra.docking.settings.BooleanSettingsDefinition; +import ghidra.docking.settings.Settings; + +/** + * Settings definition for showing function namespaces in the BSim Results table + */ +public class ShowNamespaceSettingsDefinition implements BooleanSettingsDefinition { + + public static final ShowNamespaceSettingsDefinition DEF = + new ShowNamespaceSettingsDefinition(); + + private static final boolean DEFAULT = true; + private static final String DESCRIPTION = + "Toggles showing namespace when displaying function name"; + + private static final String SHOW_NAMESPACE = "Show Namespace"; + + @Override + public boolean getValue(Settings settings) { + if (settings == null) { + return DEFAULT; + } + String value = settings.getString(SHOW_NAMESPACE); + if (value == null) { + return DEFAULT; + } + return Boolean.parseBoolean(value); + } + + @Override + public String getValueString(Settings settings) { + return Boolean.toString(getValue(settings)); + } + + @Override + public void setValue(Settings settings, boolean value) { + settings.setString(SHOW_NAMESPACE, Boolean.toString(value)); + } + + @Override + public void copySetting(Settings srcSettings, Settings destSettings) { + String value = srcSettings.getString(SHOW_NAMESPACE); + if (value == null) { + destSettings.clearSetting(SHOW_NAMESPACE); + } + else { + destSettings.setString(SHOW_NAMESPACE, value); + } + } + + @Override + public void clear(Settings settings) { + settings.clearSetting(SHOW_NAMESPACE); + } + + @Override + public String getDescription() { + return DESCRIPTION; + } + + @Override + public String getName() { + return SHOW_NAMESPACE; + } + + @Override + public String getStorageKey() { + return SHOW_NAMESPACE; + } + + @Override + public boolean hasValue(Settings settings) { + return settings.getValue(SHOW_NAMESPACE) != null; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/AbstractBSimApplyTask.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/AbstractBSimApplyTask.java new file mode 100755 index 0000000000..8ee9624743 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/AbstractBSimApplyTask.java @@ -0,0 +1,236 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results.apply; + +import java.net.MalformedURLException; +import java.net.URL; +import java.util.*; + +import docking.DockingWindowManager; +import ghidra.app.services.ProgramManager; +import ghidra.features.bsim.gui.search.results.*; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.address.Address; +import ghidra.program.model.address.AddressSpace; +import ghidra.program.model.listing.*; +import ghidra.program.util.ProgramTask; +import ghidra.util.Msg; +import ghidra.util.Swing; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +/** + * Generic task for applying information from a function match to the queried function + */ +public abstract class AbstractBSimApplyTask extends ProgramTask { + + private ServiceProvider serviceProvider; + + private List applyResults = new ArrayList<>(); + + private List resultsToBeApplied; + private FunctionManager functionManager; + private ProgramManager programManager; + private Set openedPrograms = new HashSet<>(); + + private String taskName; + + public AbstractBSimApplyTask(Program program, String taskName, List results, + ServiceProvider serviceProvider) { + super(program, "Apply Function Names", true, true, true); + this.taskName = taskName; + this.serviceProvider = serviceProvider; + functionManager = program.getFunctionManager(); + programManager = serviceProvider.getService(ProgramManager.class); + this.resultsToBeApplied = results; + } + + @Override + public void doRun(TaskMonitor monitor) { + if (programManager == null) { + Msg.error(this, "Program Manager Service not found!"); + return; + } + try { + applyResults(monitor); + releaseOpenedPrograms(); + } + catch (CancelledException e) { + // user cancelled + } + + Swing.runLater(() -> displayTaskResults(monitor.isCancelled())); + } + + protected void releaseOpenedPrograms() { + for (Program p : openedPrograms) { + p.release(this); + } + } + + private void applyResults(TaskMonitor monitor) throws CancelledException { + + Map> map = groupResultsBySearchAddress(monitor); + + monitor.initialize(map.size(), "Applying " + taskName + "..."); + + for (Address address : map.keySet()) { + applyResultsForAddress(address, map.get(address), monitor); + monitor.increment(); + } + } + + // groups results by the address of the function to be changed + private Map> groupResultsBySearchAddress(TaskMonitor monitor) + throws CancelledException { + monitor.initialize(resultsToBeApplied.size(), "Grouping results..."); + Map> map = new HashMap<>(); + for (BSimMatchResult result : resultsToBeApplied) { + monitor.increment(); + Address address = result.getAddress(); + List list = map.computeIfAbsent(address, k -> new ArrayList<>()); + list.add(result); + } + return map; + } + + private void applyResultsForAddress(Address address, List resultsForAddress, + TaskMonitor monitor) { + + Function targetFunction = functionManager.getFunctionAt(address); + if (targetFunction == null) { + error("Can't find original function", resultsForAddress.get(0)); + markRows(resultsForAddress, BSimResultStatus.ERROR); + return; + } + + List sourceFunctions = getSourceFunctions(resultsForAddress); + if (sourceFunctions.isEmpty()) { + markRows(resultsToBeApplied, BSimResultStatus.ERROR); + return; + } + + if (sourceFunctions.size() > 1) { + if (!hasSameApplyData(sourceFunctions)) { + applyResults.add(new BSimApplyResult(targetFunction.getName(), + "", BSimResultStatus.ERROR, targetFunction.getEntryPoint(), + "Attempted to apply different " + taskName + "s to the same function")); + markRows(resultsForAddress, BSimResultStatus.ERROR); + return; + } + } + + BSimApplyResult applyResult = apply(targetFunction, sourceFunctions.get(0)); + applyResults.add(applyResult); + markRows(resultsForAddress, applyResult.getStatus()); + } + + private List getSourceFunctions(List resultsForAddress) { + List functions = new ArrayList<>(); + for (BSimMatchResult bSimMatchResult : resultsForAddress) { + Function remoteFunction = getRemoteFunction(bSimMatchResult); + if (remoteFunction != null) { + functions.add(remoteFunction); + } + } + return functions; + } + + private Function getRemoteFunction(BSimMatchResult result) { + Program remoteProgram = getRemoteProgram(result); + if (remoteProgram == null) { + return null; + } + + FunctionDescription matchDescription = result.getMatchFunctionDescription(); + long addressOffset = matchDescription.getAddress(); + AddressSpace space = remoteProgram.getAddressFactory().getDefaultAddressSpace(); + Address address = space.getAddress(addressOffset); + FunctionManager remoteFunctionManager = remoteProgram.getFunctionManager(); + Function matchFunction = remoteFunctionManager.getFunctionAt(address); + + if (matchFunction == null) { + error("Couldn't find remote function at address " + address + " in remote program " + + remoteProgram.getName(), result); + } + return matchFunction; + } + + private Program getRemoteProgram(BSimMatchResult result) { + URL url = getRemoteProgramURL(result); + if (url == null) { + return null; + } + + Program remoteProgram = programManager.openCachedProgram(url, this); + if (remoteProgram == null) { + error("Open remote program failed: " + url, result); + return null; + } + + if (!openedPrograms.add(remoteProgram)) { + // The program manager added 'this' as a consumer. We previously opened it and we don't + // want the program to have the same consumer twice. + remoteProgram.release(this); + } + return remoteProgram; + } + + private void markRows(List list, BSimResultStatus state) { + for (BSimMatchResult row : list) { + row.setStatus(state); + } + } + + private URL getRemoteProgramURL(BSimMatchResult result) { + String urlString = result.getExecutableURLString(); + try { + return new URL(urlString); + } + catch (MalformedURLException e) { + error("Bad URL: " + urlString, result); + } + return null; + } + + private void displayTaskResults(boolean cancelled) { + if (!hasErrorsOrIgnores()) { + return; + } + + BSimApplyResultsDisplayDialog resultsPanel = + new BSimApplyResultsDisplayDialog(serviceProvider, applyResults, program); + DockingWindowManager.showDialog(resultsPanel); + } + + private boolean hasErrorsOrIgnores() { + for (BSimApplyResult result : applyResults) { + if (result.isError() || result.isIgnored()) { + return true; + } + } + return false; + } + + private void error(String message, BSimMatchResult result) { + applyResults.add(new BSimApplyResult(result, BSimResultStatus.ERROR, message)); + } + + protected abstract boolean hasSameApplyData(List functions); + + protected abstract BSimApplyResult apply(Function target, Function source); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/NameAndNamespaceBSimApplyTask.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/NameAndNamespaceBSimApplyTask.java new file mode 100644 index 0000000000..b8ce44ced9 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/NameAndNamespaceBSimApplyTask.java @@ -0,0 +1,74 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results.apply; + +import java.util.List; + +import ghidra.features.bsim.gui.search.results.*; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.program.model.symbol.SymbolUtilities; +import ghidra.program.util.FunctionUtility; + +/** + * Task for applying names and namespaces from a match function to the queried function + */ +public class NameAndNamespaceBSimApplyTask extends AbstractBSimApplyTask { + + public NameAndNamespaceBSimApplyTask(Program program, List results, + ServiceProvider serviceProvider) { + super(program, "Function Name", results, serviceProvider); + } + + @Override + protected boolean hasSameApplyData(List functions) { + String name = functions.get(0).getName(true); + for (int i = 1; i < functions.size(); i++) { + if (!functions.get(i).getName(true).equals(name)) { + return false; + } + } + return true; + } + + @Override + protected BSimApplyResult apply(Function target, Function source) { + String defaultFunctionName = SymbolUtilities.getDefaultFunctionName(source.getEntryPoint()); + if (defaultFunctionName.equals(source.getName())) { + return new BSimApplyResult(target, source, BSimResultStatus.ERROR, + "Can't apply default function names"); + } + String targetFullName = target.getName(true); + String sourceFullName = source.getName(true); + + if (targetFullName.equals(sourceFullName)) { + return new BSimApplyResult(target, source, BSimResultStatus.IGNORED, + "Functions already have the same name"); + } + + try { + FunctionUtility.applyNameAndNamespace(target, source); + } + catch (Exception e) { + return new BSimApplyResult(target, source, BSimResultStatus.ERROR, + "Rename failed (" + e.getMessage() + ")"); + } + + return new BSimApplyResult(target, source, BSimResultStatus.NAME_APPLIED, ""); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/SignatureBSimApplyTask.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/SignatureBSimApplyTask.java new file mode 100644 index 0000000000..ba29534bf6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/gui/search/results/apply/SignatureBSimApplyTask.java @@ -0,0 +1,69 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results.apply; + +import java.util.List; + +import ghidra.features.bsim.gui.search.results.*; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.data.DataTypeConflictHandler; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.SymbolUtilities; +import ghidra.program.util.FunctionUtility; + +/** + * Task for applying names, namespaces, and signatures from a match function to the queried function + */ +public class SignatureBSimApplyTask extends AbstractBSimApplyTask { + + private boolean applyEmptyStructures; + + public SignatureBSimApplyTask(Program program, List results, + boolean applyEmptyStructures, ServiceProvider serviceProvider) { + super(program, "Function Signature", results, serviceProvider); + this.applyEmptyStructures = applyEmptyStructures; + } + + @Override + protected boolean hasSameApplyData(List functions) { + FunctionSignature firstSignature = functions.get(0).getSignature(false); + for (int i = 1; i < functions.size(); i++) { + FunctionSignature signature = functions.get(i).getSignature(false); + if (!firstSignature.isEquivalentSignature(signature)) { + return false; + } + } + return true; + } + + @Override + protected BSimApplyResult apply(Function target, Function source) { + String defaultFunctionName = SymbolUtilities.getDefaultFunctionName(source.getEntryPoint()); + if (defaultFunctionName.equals(source.getName())) { + return new BSimApplyResult(target, source, BSimResultStatus.ERROR, + "Can't apply default function names"); + } + try { + FunctionUtility.applySignature(target, source, applyEmptyStructures, + DataTypeConflictHandler.REPLACE_EMPTY_STRUCTS_OR_RENAME_AND_ADD_HANDLER); + return new BSimApplyResult(target, source, BSimResultStatus.SIGNATURE_APPLIED, ""); + } + catch (Exception e) { + return new BSimApplyResult(target, source, BSimResultStatus.ERROR, + "Apply signature failed (" + e.getMessage() + ")"); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimClientFactory.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimClientFactory.java new file mode 100755 index 0000000000..7b207d34ee --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimClientFactory.java @@ -0,0 +1,160 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.net.MalformedURLException; +import java.net.URL; + +import ghidra.features.bsim.query.client.PostgresFunctionDatabase; +import ghidra.features.bsim.query.elastic.ElasticDatabase; +import ghidra.features.bsim.query.file.H2FileFunctionDatabase; +import ghidra.framework.protocol.ghidra.GhidraURL; + +public class BSimClientFactory { + + /** + * Build a root URL for connecting to a BSim database. + * 1) A valid protocol must be provided. + * 2) There must be a path of exactly 1 element, which names the specific repository + * Acceptable protocols are postgresql:// https://, (or possibly http://) file:/ + * + * @param urlString the URL to build + * @return the parsed URL object + * @throws MalformedURLException if the URL string cannot be parsed + */ + public static URL buildURL(String urlString) throws MalformedURLException { + URL url = new URL(urlString); + checkBSimServerURL(url); + return url; + } + + /** + * Validate BSim DB URL. + * Acceptable protocols are postgresql:// https://, (or possibly http://) file:/ + * @param url BSim DB URL + * @throws MalformedURLException if the URL string is not a support BSim DB URL + */ + public static void checkBSimServerURL(URL url) throws MalformedURLException { + String protocol = url.getProtocol(); + if (!protocol.equals("postgresql") && !protocol.equals("https") && + !protocol.equals("elastic") && !protocol.equals("file")) { + throw new MalformedURLException("Protocol not permissable for BSim URL"); + } + String path = url.getPath(); + if (path == null || path.length() == 0 || path.equals("/")) { + throw new MalformedURLException("BSim URL missing DB name/path"); + } + if (!"file".equals(protocol) && path.indexOf('/', 1) >= 0) { + throw new MalformedURLException("BSim URL must specify exactly 1 path element"); + } + } + + /** + * Construct the root URL to a specific BSim repository given a "related" URL. + * The root URL will have an explicit protocol, a hostname + other mods (the authority), and 1 level of path + * this first level path indicates the particular repository being referenced on the host. + * The "related" URL -url- can be an explicitly provided URL pointing to the BSim repository, + * possibly with additional path levels, which are simply stripped from the final root URL. + * Alternately -url- can reference a ghidra server, as indicated by the "ghidra" protocol. + * In this case the true BSim URL is derived from ghidra URL in some way + * @param urlString is the "related" URL + * @return the root BSim URL + * @throws MalformedURLException if the given URL string cannot be parsed + * @throws IllegalArgumentException if local ghidra URL is specified + */ + public static URL deriveBSimURL(String urlString) + throws IllegalArgumentException, MalformedURLException { + URL url = new URL(urlString); // URL used only for parsing purposes + String protocol = url.getProtocol(); + if ("postgresql".equals(protocol) || "https".equals(protocol) || + "elastic".equals(protocol) || "file".equals(protocol)) { + checkBSimServerURL(url); + return url; // URL already corresponds to BSim server protocol + } + if (!GhidraURL.isServerRepositoryURL(url)) { + throw new IllegalArgumentException("Unable to infer BSim URL from: " + url); + } + String path = url.getPath(); // Get the full path of the URL + if (path == null || path.length() == 0 || path.equals("/")) { // There must always be some kind of path, so we can derive the repository + throw new MalformedURLException("URL is missing a repository path"); + } + int endrepos = path.indexOf('/', 1); // Find the end of the first level of the path + String repositoryURL; + if (url.getProtocol().equals(GhidraURL.PROTOCOL)) { // Is this a ghidra URL + // TODO: we could set things up so that a ghidra server could be queried for its associated BSim server + // "ghidra://host/repo?service=bsim" + // String repositoryURL = "ghidra://" + ghidraURL.getAuthority() + "?service=bsim"; + + // Currenly all we do is assume that the BSim server is a PostgreSQL server + // on the same host and with the same repo name as the ghidra server + repositoryURL = "postgresql://" + url.getHost(); // Just use the hostname + } + else { + // For all other URL forms, we assume we are being handed the protocol and hostname (authority) + // explicitly. We keep them for the final BSim URL + repositoryURL = url.getProtocol() + "://" + url.getAuthority(); + } + // Attach the first level of the path, which indicates the repository + if (endrepos < 0) { + repositoryURL = repositoryURL + path; + } + else { + repositoryURL = repositoryURL + path.substring(0, endrepos); + } + return buildURL(repositoryURL); + } + + /** + * Given the URL for a BSim server construct the appropriate BSim client object (implementing FunctionDatabase) + * @param bsimServerInfo BSim server details + * @param async true if database commits should be asynchronous + * @return the database client + */ + public static FunctionDatabase buildClient(BSimServerInfo bsimServerInfo, boolean async) { + try { + return buildClient(bsimServerInfo.toURL(), async); + } + catch (MalformedURLException e) { + throw new RuntimeException(e); // unexpected + } + } + + /** + * Given the URL for a BSim server construct the appropriate BSim client object + * (implementing FunctionDatabase). Returned instance must be + * {@link FunctionDatabase#close() closed} when done using it to prevent depletion + * of database connections. + * @param bsimURL URL supplied by the user + * @param async true if database commits should be synchronous + * @return the database client + * @throws MalformedURLException if there's a problem creating the elastic database + */ + public static FunctionDatabase buildClient(URL bsimURL, boolean async) + throws MalformedURLException { + + String protocol = bsimURL.getProtocol(); + if (protocol.equals("postgresql")) { + return new PostgresFunctionDatabase(bsimURL, async); + } + if (protocol.equals("https") || protocol.equals("elastic")) { + return new ElasticDatabase(bsimURL); + } + if ("file".equals(protocol)) { + return new H2FileFunctionDatabase(bsimURL); + } + throw new MalformedURLException("Unsupported protocol: " + protocol); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimControlLaunchable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimControlLaunchable.java new file mode 100755 index 0000000000..d5f7cc7c06 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimControlLaunchable.java @@ -0,0 +1,1444 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.io.*; +import java.net.Authenticator; +import java.net.InetAddress; +import java.nio.file.Files; +import java.nio.file.attribute.PosixFilePermissions; +import java.security.*; +import java.security.KeyStore.PasswordProtection; +import java.sql.*; +import java.util.*; + +import javax.naming.ldap.LdapName; +import javax.naming.ldap.Rdn; +import javax.security.auth.DestroyFailedException; + +import org.xml.sax.ErrorHandler; +import org.xml.sax.SAXException; + +import ghidra.GhidraApplicationLayout; +import ghidra.GhidraLaunchable; +import ghidra.features.bsim.query.ingest.BSimLaunchable; +import ghidra.framework.*; +import ghidra.framework.client.ClientUtil; +import ghidra.net.ApplicationKeyManagerUtils; +import ghidra.util.Msg; +import ghidra.util.exception.AssertException; +import ghidra.util.task.TaskMonitor; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; +import utilities.util.FileUtilities; + +public class BSimControlLaunchable implements GhidraLaunchable { + + public static final String PORT_OPTION = "port="; + public static final String CAFILE_OPTION = "cafile="; + public static final String AUTH_OPTION = "auth="; + public static final String DN_OPTION = "dn="; + public static final String CERT_OPTION = "cert="; + public static final String NO_LOCAL_AUTH_OPTION = "--noLocalAuth"; + public static final String USER_OPTION = "user="; + public static final String FORCE_OPTION = "--force"; + + private final static String START_COMMAND = "start"; + private final static String STOP_COMMAND = "stop"; + private final static String PASSWORD_COMMAND = "resetpassword"; + private final static String PRIVILEGE_COMMAND = "changeprivilege"; + private final static String ADDUSER_COMMAND = "adduser"; + private final static String DROPUSER_COMMAND = "dropuser"; + private final static String RESET_COMMAND = "changeauth"; + private final static String POSTGRES = "postgresql"; + private final static String POSTGRES_BUILD_SCRIPT = "Ghidra/Features/BSim/make-postgres.sh"; + private final static String POSTGRES_CONFIGFILE = "postgresql.conf"; + private final static String POSTGRES_CONNECTFILE = "pg_hba.conf"; + private final static String POSTGRES_IDENTFILE = "pg_ident.conf"; + private final static String POSTGRES_ROOTCA = "root.crt"; + private final static String PASSWORD_METHOD = "scram-sha-256"; + private final static String TRUST_METHOD = "trust"; + private final static String CERTIFICATE_METHOD = "cert"; + private final static String CERTIFICATE_OPTIONS = "map=mymap clientcert=verify-full"; +// private final static String CERTIFICATE_OPTIONS = "map=mymap clientcert=1"; // For PKI certificates prior to PostgreSQL 12 + private final static String POSTGRES_MAP_IDENTIFIER = "mymap"; + private final static String DEFAULT_PASSWORD = "changeme"; + private final static int AUTHENTICATION_NONE = 0; + private final static int AUTHENTICATION_PASSWORD = 1; + private final static int AUTHENTICATION_PKI = 2; + private GhidraApplicationLayout layout = null; // For holding on to JDBC logger so we can filter messages + private File dataDirectory; // Directory containing postgres datafiles + private File postgresRoot; // Directory containing postgres software + private File postgresControl; // "pg_ctl" utility within postgres software + private File certAuthorityFile = null; // Certificate authority file provided by the user + private String certParameter = null; // Path to certificate provided by user + private String distinguishedName = null; // Certificate distinguished name provided by the user + private String commonName = null; // Common name extracted from distinguishedName + private String connectingUserName = null; // User-name used to establish connection + private String specifiedUserName = null; // -username- (add/drop) operation is being performed on + private boolean adminPrivilegeRequested = false; // true is attempting to give user admin privileges + private boolean forceShutdown = false; // Whether or not to force a shutdown (--force) + private String loadLibraryVar = null; // Environment variable pointing to postgres shared libraries + private String loadLibraryValue = null; // Directory containing shared libraries within postgres software + private int port = -1; // Port over which to connect to postgres server, (-1 indicates default port is used) + private int localAuthentication = AUTHENTICATION_NONE; // Type of authentication required for local connections + private int hostAuthentication = AUTHENTICATION_NONE; // Type of authentication for remote connections + private boolean authConfigPresent = false; // True if the [auth=..] option or the [--noLocalAuth] is present + private File passwordFile = null; // File containing newly established password + private char[] adminPasswordData = null; // Password data being sent to postgres server for authentication + + // Database connection that can be persisted so we don't need to recreate one + // for every call. + private Connection localConnection; + + /** + * Exception triggered by missing, unknown, or improperly formatted command-line arguments + */ + public static class ArgumentException extends Exception { + public ArgumentException(String message) { + super(message); + } + } + + /** + * Class for processing standard output or standard error for processes invoked by BSimControl + * The streams can be optionally suppressed or dumped to System.out + */ + private class IOThread extends Thread { + private BufferedReader shellOutput; // Reader for the particular output stream + private boolean suppressOutput; // If false, shell output is printed on the console + + public IOThread(InputStream input, boolean suppressOut) { + shellOutput = new BufferedReader(new InputStreamReader(input)); + suppressOutput = suppressOut; + } + + @Override + public void run() { + String line = null; + try { + while ((line = shellOutput.readLine()) != null) { + if (!suppressOutput) { + System.out.println(line); + } + } + } + catch (Exception e) { + // DO NOT USE LOGGING HERE (class loader) + System.err.println("Unexpected Exception: " + e.getMessage()); + e.printStackTrace(System.err); + } + } + } + + public BSimControlLaunchable() { + // Constructor for main launcher + } + + /** + * Verify that the given file is a PEM certificate + * @param testFile the file to test + * @return true if testFile looks like a PEM certificate + * @throws IOException if there is a problem reading the given file + */ + private static boolean verifyPEMFormat(File testFile) throws IOException { + BufferedReader reader = new BufferedReader(new FileReader(testFile)); + try { + // All we currently do is search for the certificate header in the first 200 lines + for (int i = 0; i < 200; ++i) { + String line = reader.readLine(); + if (line == null) { + break; + } + if (line.startsWith(ApplicationKeyManagerUtils.BEGIN_CERT)) { + return true; + } + } + } + finally { + reader.close(); + } + return false; + } + + /** + * Parse the -distinguishedName- String, verifying it is has the correct format for a + * X509 certificate distinguished name. Try to extract the common name portion of the + * distinguished name and assign it to -commonName- + * @throws ArgumentException if the distinguished name is improperly formatted or the common name is missing + */ + private void validateDistinguishedName() throws ArgumentException { + commonName = null; + try { + LdapName ldapName = new LdapName(distinguishedName); + for (Rdn rdn : ldapName.getRdns()) { + if (rdn.getType().equalsIgnoreCase("CN")) { + commonName = rdn.getValue().toString(); + break; + } + } + if (commonName == null) { + throw new ArgumentException("Missing common name attribute"); + } + } + catch (Exception e) { + throw new ArgumentException("Improperly formatted distinguished name"); + } + } + + /** + * @return true if the server (referred to by -postgresRoot-) is running + * @throws IOException if there is a problem running the command + * @throws InterruptedException if there is a problem running the command + */ + private boolean isServerRunning() throws IOException, InterruptedException { + File createCommand = new File(postgresRoot, "bin/pg_isready"); + List command = new ArrayList(); + command.add(createCommand.getAbsolutePath()); + if ((port != -1) && (port != 5432)) { // Non-default port + command.add("-p"); + command.add(Integer.toString(port)); + } + int ret = runCommand(null, command, loadLibraryVar, loadLibraryValue); + return (ret == 0); + } + + private char[] requestPassword(String prompt) { + String host = "localhost"; + InetAddress addr = InetAddress.getLoopbackAddress(); + String protocol = "postgresql"; + String scheme = "NO_NAME"; + return Authenticator + .requestPasswordAuthentication(host, addr, port, protocol, prompt, scheme) + .getPassword(); + } + + /** + * (For a new postgres server) Establish an administrative password, by requesting the password + * from the user, and then having the user re-enter the password. The password is stored in + * the character array -adminPasswordData- and written to the file -passwordFile- + * for access by the postgres "init" process + * @throws IOException if there is a problem obtaining the password + */ + private void establishAdminPassword() throws IOException { + for (;;) { + adminPasswordData = requestPassword("Set admin(" + connectingUserName + ") password:"); + if (adminPasswordData == null) { + throw new IOException("Unable to obtain password"); + } + char[] repeatPass = requestPassword("Please re-enter password:"); + boolean match = comparePasswordData(adminPasswordData, repeatPass); + clearPasswordData(repeatPass); + if (match) { + break; + } + cleanupPasswordData(); + System.out.println("Passwords do not match"); + } + passwordFile = Files + .createTempFile("bsim", ".dat", + PosixFilePermissions + .asFileAttribute(PosixFilePermissions.fromString("rw-------"))) + .toFile(); + FileWriter writer = new FileWriter(passwordFile); + writer.write(adminPasswordData); + writer.close(); + } + + /** + * Clear (sensitive) data for a particular character array so it is no longer accessible from the heap + * @param password is the array of sensitive characters + */ + private static void clearPasswordData(char[] password) { + if (password != null) { + for (int i = 0; i < password.length; ++i) { + password[i] = ' '; + } + } + } + + /** + * Compare that two character arrays contain exactly the same data + * @param password password to compare + * @param repeatPass password to compare + * @return true if the character sequences are non-null and identical + */ + private static boolean comparePasswordData(char[] password, char[] repeatPass) { + if (password == null || repeatPass == null) { + return false; + } + if (password.length != repeatPass.length) { + return false; + } + for (int i = 0; i < repeatPass.length; ++i) { + if (repeatPass[i] != password[i]) { + return false; + } + } + return true; + } + + /** + * Make sure password data, stored either in the heap or in a temporary file, is scrubbed + * @throws IOException if the password file cannot be deleted + */ + private void cleanupPasswordData() throws IOException { + if (passwordFile != null) { + if (!passwordFile.delete()) { + throw new IOException( + "Unable to delete password file: " + passwordFile.getAbsolutePath()); + } + passwordFile = null; + } + clearPasswordData(adminPasswordData); + adminPasswordData = null; + } + + /** + * Servers that allow SSL connections are required to have a certificate that allows it to + * authenticate itself to users. The BSim server does not authenticate itself to clients, but + * a certificate must still be present. We generate a self-signed certificate. + * @param certFile will hold the public portion of the generated certificate + * @param passFile will hold the private portion + * @throws IOException if the password file cannot be opened for writing + * @throws GeneralSecurityException if the keystore cannot be created + */ + private void generateSelfSignedCertificate(File certFile, File passFile) + throws IOException, GeneralSecurityException { + + String alias = "bsimroot"; + char[] password = "unusedpassword".toCharArray(); + + PasswordProtection pp = new PasswordProtection(password); + try { + // TODO: should subjectAlternativeNames be supported? + KeyStore keyStore = ApplicationKeyManagerUtils.createKeyStore(alias, "CN=BSimServer", + 365 * 2, null, null, "JKS", null, password); + + ApplicationKeyManagerUtils.exportX509Certificates(keyStore.getCertificateChain(alias), + certFile); + Key key = keyStore.getKey(alias, password); + + try (FileOutputStream fout = new FileOutputStream(passFile); + PrintWriter writer = new PrintWriter(fout)) { + writer.print("-----BEGIN PRIVATE KEY-----"); + writer.println(); + String base64 = Base64.getEncoder().encodeToString(key.getEncoded()); + while (base64.length() != 0) { + int endIndex = Math.min(44, base64.length()); + String line = base64.substring(0, endIndex); + writer.println(line); + base64 = base64.substring(endIndex); + } + writer.println("-----END PRIVATE KEY-----"); + writer.println(); + } + + passFile.setExecutable(false, false); // Clear execute permission for everybody + passFile.setReadable(false, false); // Clear read permission for everybody + passFile.setWritable(false, false); // Clear write permission for everybody + passFile.setReadable(true, true); // Let owner read the file + } + catch (NoSuchAlgorithmException | UnrecoverableEntryException e) { + throw new KeyStoreException("Failed to generate BSim server certificate", e); + } + finally { + Arrays.fill(password, ' '); + try { + pp.destroy(); + } + catch (DestroyFailedException e) { + throw new AssertException(e); // unexpected for simple password clearing + } + } + } + + /** + * Initialize enough of Ghidra to allow navigation of configuration files and to allow SSL connections + * @throws IOException if the headless authenticator cannot be initialized + * @throws ClassNotFoundException if the postgres driver class cannot be found + */ + private void initializeApplication() throws IOException, ClassNotFoundException { + if (layout != null) { + // Initialize application environment consistent with bsim command + BSimLaunchable.initializeApplication(layout, 0, connectingUserName, certParameter); + } + } + + /** + * Create a local connection to a postgres server. A full SSL connection is created using + * Ghidra's infrastructure. If the initial connection fails because password authentication + * was requested, collect the administrative password from the user, and try the connection again + * @return the established connection object. Respect any command-line "port= .." option. + * @throws SQLException if the db connection cannot be established + * @throws IOException if the user password cannot be retrieved + */ + private Connection createLocalConnection() throws SQLException, IOException { + Properties properties = new Properties(); + properties.setProperty("sslmode", "require"); + properties.setProperty("sslfactory", "ghidra.net.ApplicationSSLSocketFactory"); + properties.setProperty("user", connectingUserName); + StringBuilder buffer = new StringBuilder(); + buffer.append("jdbc:postgresql://localhost"); + if ((port != -1) && (port != 5432)) { // Non-default port + buffer.append(':'); + buffer.append(port); + } + buffer.append("/template1"); + String connstring = buffer.toString(); + if (adminPasswordData == null) { + try { + Connection pdb = DriverManager.getConnection(connstring, properties); + return pdb; + } + catch (SQLException e) { + if (!e.getMessage().contains("password-based authentication") && + !e.getMessage().contains("SCRAM-based authentication")) { + throw e; + } + } + adminPasswordData = requestPassword("User " + connectingUserName + " password:"); + if (adminPasswordData == null) { + throw new IOException("Unable to obtain password"); + } + } + String passString = new String(adminPasswordData); + properties.setProperty("password", passString); + return DriverManager.getConnection(connstring, properties); // Try again providing driver a password + } + + /** + * Execute SQL statement on a connection that returns nothing. + * If execution fails, the connection is closed before throwing exception + * @param pdb is the connection + * @param statementString is the SQL statement + * @throws SQLException if the sql statement cannot be created or executed + */ + private static void executeSQLStatement(Connection pdb, String statementString) + throws SQLException { + try (Statement st = pdb.createStatement()) { + st.executeUpdate(statementString); + } + catch (SQLException err) { + pdb.close(); + throw err; + } + } + + /** + * On a running server, establish a local connection and enable the BSim specific extension for that server + * @throws SQLException if the sql statement cannot be executed + * @throws IOException if the db connection cannot be established + */ + private void enableLSHExtension() throws SQLException, IOException { + Connection pdb = createLocalConnection(); + + try { + executeSQLStatement(pdb, "CREATE EXTENSION IF NOT EXISTS lshvector"); + } + finally { + pdb.close(); + } + + } + + /** + * Invoke an external executable/command, display the output and error streams on the console, + * the exit condition of the command is returned. If the exit condition indicates and error, + * but a line of the error stream matches a provided String, the error is suppressed + * @param directory is the working directory for the command + * @param command is the command-line (including arguments) + * @param envvar if non-null, is an environment variable to set for the command + * @param value is the corresponding environment variable value + * @return the exit status of the command (0=no error) + * @throws IOException if the process cannot be started + * @throws InterruptedException if there is a problem waiting for the process to finish + */ + private int runCommand(File directory, List command, String envvar, String value) + throws IOException, InterruptedException { + ProcessBuilder processBuilder = new ProcessBuilder(command); + processBuilder.directory(directory); // Set the working directory + if (envvar != null) { + Map environment = processBuilder.environment(); + environment.put(envvar, value); + } + Process process = processBuilder.start(); + + new IOThread(process.getInputStream(), true).start(); + IOThread errThread = new IOThread(process.getErrorStream(), true); + errThread.start(); + + int retval = process.waitFor(); + return retval; + } + + /** + * Tune the postgres configuration and authentication files (postgresql.conf and pg_hba.conf) + * based on the command-line options passed in by the user and the ghidra specific configuration options + * @param inputFile is the unmodified postgresql.conf file + * @param outputFile will hold the new modified version of postgresql.conf + * @param inHbaFile is the original pg_hba.conf file + * @param outHbaFile will hold the new modified version of pg_hba.conf + * @param serverConfigFile is the XML file holding ghidra specific BSim configuration options + * @throws SAXException if the xml pull parser cannot be created + * @throws IOException if the authentication fails + */ + private void tuneConfig(File inputFile, File outputFile, File inHbaFile, File outHbaFile, + File serverConfigFile) throws SAXException, IOException { + ErrorHandler handler = SpecXmlUtils.getXmlHandler(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(serverConfigFile, handler, false); + ServerConfig serverConfig = new ServerConfig(); + serverConfig.restoreXml(parser); + if ((port != -1) && (port != 5432)) { + serverConfig.addKey("port", Integer.toString(port)); + } + if (localAuthentication == AUTHENTICATION_NONE) { + serverConfig.setLocalAuthentication(TRUST_METHOD, null); + } + else if (localAuthentication == AUTHENTICATION_PASSWORD) { + serverConfig.setLocalAuthentication(PASSWORD_METHOD, null); + } + else if (localAuthentication == AUTHENTICATION_PKI) { + serverConfig.setLocalAuthentication(CERTIFICATE_METHOD, CERTIFICATE_OPTIONS); + } + else { + throw new IOException("Unsupported local authentication type"); + } + if (hostAuthentication == AUTHENTICATION_NONE) { + serverConfig.setHostAuthentication(TRUST_METHOD, null); + } + else if (hostAuthentication == AUTHENTICATION_PASSWORD) { + serverConfig.setHostAuthentication(PASSWORD_METHOD, null); + } + else if (hostAuthentication == AUTHENTICATION_PKI) { + serverConfig.setHostAuthentication(CERTIFICATE_METHOD, CERTIFICATE_OPTIONS); + } + else { + throw new IOException("Unsupported host authentication type"); + } + if (hostAuthentication == AUTHENTICATION_PKI || localAuthentication == AUTHENTICATION_PKI) { + serverConfig.addKey("ssl_ca_file", '\'' + POSTGRES_ROOTCA + '\''); // Turn on certificate authority + } + serverConfig.patchConfig(inputFile, outputFile); + serverConfig.patchConnect(inHbaFile, outHbaFile); + } + + /** + * Set-up the PostgreSQL shared library environment variable for this Ghidra installation + */ + private void setupPostgresSharedLibrary() { + if (Platform.CURRENT_PLATFORM.getOperatingSystem() == OperatingSystem.MAC_OS_X) { + loadLibraryVar = "DYLD_LIBRARY_PATH"; + } + else { + loadLibraryVar = "LD_LIBRARY_PATH"; + } + File postgresLibrary = new File(postgresRoot, "lib"); + loadLibraryValue = postgresLibrary.getAbsolutePath(); + } + + /** + * Locate the "pg_ctl" executable within the PostgreSQL installation. Unpack the installation + * if it is not already. + * @throws IOException if postgres folder cannot be determined + */ + private void discoverPostgresInstall() throws IOException { + try { + postgresRoot = Application.getOSFile(POSTGRES); // find PostgreSQL build for os + postgresControl = new File(postgresRoot, "bin/pg_ctl"); + if (!postgresControl.isFile()) { + throw new IOException("PostgreSQL pg_ctl command not found: " + postgresControl); + } + setupPostgresSharedLibrary(); + } + catch (OSFileNotFoundException e) { + throw new IOException("PostgreSQL not found and must be built (see " + + POSTGRES_BUILD_SCRIPT + ", view script for details)"); + } + } + + /** + * Recover the parameter settings from a previously initialized server + * @param configFile is main configuration file: port, adminPassword, hostAuthentication + * @param hbaFile is the connection file + * @throws IOException if the server port is invalid + */ + private void recoverConfigurationParameters(File configFile, File hbaFile) throws IOException { + ServerConfig serverConfig = new ServerConfig(); + serverConfig.addKey("port", ""); + serverConfig.scanConfig(configFile); + String value = serverConfig.getValue("port"); + int scannedPort = 5432; + if (value.length() != 0) { + scannedPort = Integer.parseInt(value); + } + if (port != -1 && (scannedPort != port)) { + throw new IOException("Server is configured to run on port " + + Integer.toString(scannedPort) + ": Change in " + POSTGRES_CONFIGFILE); + } + port = scannedPort; + + serverConfig.scanConnect(hbaFile); + String localMethod = serverConfig.getLocalAuthentication(); + if (localMethod == null || localMethod.equals(TRUST_METHOD)) { + localAuthentication = AUTHENTICATION_NONE; + } + else if (localMethod.equals(PASSWORD_METHOD)) { + localAuthentication = AUTHENTICATION_PASSWORD; + } + else if (localMethod.equals(CERTIFICATE_METHOD)) { + localAuthentication = AUTHENTICATION_PKI; + } + String hostMethod = serverConfig.getHostAuthentication(); + if (hostMethod == null || hostMethod.equals(TRUST_METHOD)) { + hostAuthentication = AUTHENTICATION_NONE; + } + else if (hostMethod.equals(PASSWORD_METHOD)) { + hostAuthentication = AUTHENTICATION_PASSWORD; + } + else if (hostMethod.equals(CERTIFICATE_METHOD)) { + hostAuthentication = AUTHENTICATION_PKI; + } + } + + /** + * Make sure certificate authority needed for pki was provided by user, otherwise throw exception + * @throws IOException if the cert file is invalid + * @throws GeneralSecurityException if the cert file is not a valid certificate + */ + private void checkCertAuthorityFile() throws IOException, GeneralSecurityException { + if (certAuthorityFile == null) { + throw new IOException( + "PKI authentication requested, but certificate authority file not provided"); + } + if (!certAuthorityFile.isFile()) { + throw new IOException( + certAuthorityFile.getAbsolutePath() + " is not a valid certification authority"); + } + if (!verifyPEMFormat(certAuthorityFile)) { + throw new GeneralSecurityException( + "File " + certAuthorityFile.getName() + " does not appear to be a certificate"); + } + } + + /** + * Locate the PostgreSQL configuration and authentication files (postgresql.conf and pg_hba.conf) + * and recover the settings pertinent to BSimControl. If the data directory has not been initialized yet, + * run PostgreSQL's init command to perform the initialization and then tailor the configuration + * based on BSimControl's command-line options and the Ghidra specific configuration options + * @throws IOException if the module data file cannot be retrieved + * @throws InterruptedException if the postgres command is interrupted + * @throws SAXException if tuneConfig fails + * @throws GeneralSecurityException if the cert file cannot be processed + */ + private void initializeDataDirectory() + throws IOException, InterruptedException, SAXException, GeneralSecurityException { + File configFile = new File(dataDirectory, POSTGRES_CONFIGFILE); + File hbaFile = new File(dataDirectory, POSTGRES_CONNECTFILE); + if (configFile.exists()) { + recoverConfigurationParameters(configFile, hbaFile); + return; + } + File serverConfigFile = Application.getModuleDataFile("serverconfig.xml").getFile(false); + if (hostAuthentication == AUTHENTICATION_PKI) { + checkCertAuthorityFile(); + System.out.println("Remote client authentication with PKI certificates"); + } + else if (hostAuthentication == AUTHENTICATION_PASSWORD) { + System.out.println("Remote client authentication via password"); + } + else { + System.out.println("No client authentication"); + } + System.out.println("Initializing data directory"); + List command = new ArrayList(); + command.add(postgresControl.getAbsolutePath()); + command.add("init"); + command.add("-o"); + command.add("'--username=" + connectingUserName + '\''); + if (hostAuthentication == AUTHENTICATION_PASSWORD) { + establishAdminPassword(); + command.add("-o"); + command.add("'--pwfile=" + passwordFile.getAbsolutePath() + '\''); + } + else if (hostAuthentication == AUTHENTICATION_PKI) { + if (commonName == null) { + throw new GeneralSecurityException( + "Distinguished name required for " + connectingUserName + " (dn=\"..\")"); + } + checkCertAuthorityFile(); + } + command.add("-D"); + command.add(dataDirectory.getAbsolutePath()); + int res = runCommand(null, command, loadLibraryVar, loadLibraryValue); + if (res != 0) { + throw new IOException("Error initializing postgres database"); + } + File configCopy = new File(dataDirectory, POSTGRES_CONFIGFILE + ".orig"); + + if (hostAuthentication == AUTHENTICATION_PKI || localAuthentication == AUTHENTICATION_PKI) { + File rootCA = new File(dataDirectory, POSTGRES_ROOTCA); + FileUtilities.copyFile(certAuthorityFile, rootCA, false, null); + addCertificateName(connectingUserName); + } + + // Move the original configuration file + if (!configFile.renameTo(configCopy)) { + throw new IOException("Error copying original configuration file"); + } + + File hbaCopy = new File(dataDirectory, POSTGRES_CONNECTFILE + ".orig"); + + // Move the original connection file + if (!hbaFile.renameTo(hbaCopy)) { + throw new IOException("Error copying original connection file"); + } + // Patch the configuration + tuneConfig(configCopy, configFile, hbaCopy, hbaFile, serverConfigFile); + System.out.println("Generating servers SSL certificate"); + generateSelfSignedCertificate(new File(dataDirectory, "server.crt"), + new File(dataDirectory, "server.key")); + } + + /** + * Scan the PostgreSQL data directory from the command-line + * Make sure the directory exists and establish the File object -dataDirectory- + * @param params are the command-line options + * @param slot is the position to retrieve the data directory + * @throws ArgumentException if the data directory is invalid + * @throws IOException if the canonical file cannot be retrieved + */ + private void scanDataDirectory(String[] params, int slot) + throws ArgumentException, IOException { + if (params.length <= slot) { + throw new ArgumentException("Missing data directory"); + } + dataDirectory = new File(params[slot]); + if (!dataDirectory.isDirectory()) { + throw new ArgumentException( + "Data directory " + dataDirectory.getAbsolutePath() + " does not exist"); + } + dataDirectory = dataDirectory.getCanonicalFile(); + } + + /** + * Scan the username from the command-line + * @param params are the command-line options + * @param slot is the position to retrieve the username + * @throws ArgumentException if the user name is not in the given params + */ + private void scanUsername(String[] params, int slot) throws ArgumentException { + if (params.length <= slot) { + throw new ArgumentException("Missing username"); + } + specifiedUserName = params[slot]; + } + + /** + * Scan command-line for a particular privilege level. Administrator privileges are + * requested with the exact String "admin", anything is a request for a read-only user + * @param params are the command-line options + * @param slot is the position to retrieve the user name + * @throws ArgumentException the privilege parameter is missing + */ + private void scanPrivilege(String[] params, int slot) throws ArgumentException { + if (params.length <= slot) { + throw new ArgumentException("Missing desired privilege (admin or user)"); + } + if (params[slot].equals("admin")) { + adminPrivilegeRequested = true; + } + else if (params[slot].equals("user")) { + adminPrivilegeRequested = false; + } + else { + throw new ArgumentException("Expecting privilege option (admin or user)"); + } + } + + /** + * Start a PostgreSQL server, configured for BSim, on the local host. + * If the data directory is already populated, the server process is simply restarted. + * If the data directory is empty, a new server configuration is established, and the server is started. + * Authentication may be necessary, either via password or certificate, in order to enable + * the BSim extension on the server + * + * @throws IOException if postgres cannot be started + * @throws InterruptedException if the process fails during the run + * @throws SAXException if the data directory cannot be initialized + * @throws GeneralSecurityException if the authentication fails + */ + private void startCommand() + throws IOException, InterruptedException, SAXException, GeneralSecurityException { + discoverPostgresInstall(); + initializeDataDirectory(); + + if (localAuthentication == AUTHENTICATION_PKI && certParameter == null) { + throw new GeneralSecurityException( + "Path to certificate necessary to start server (cert=/path/to/cert)"); + } + File logFile = new File(dataDirectory, "logfile"); + List command = new ArrayList(); + command.add(postgresControl.getAbsolutePath()); + command.add("start"); + command.add("-w"); + command.add("-D"); + command.add(dataDirectory.getAbsolutePath()); + command.add("-l"); + command.add(logFile.getAbsolutePath()); + int res = runCommand(null, command, loadLibraryVar, loadLibraryValue); + if (res != 0) { + throw new IOException("Could not start postgres server process"); + } + + System.out.println("Server started"); + boolean extensionEnabled = true; + try { + enableLSHExtension(); + } + catch (SQLException e) { + System.out.println(e.getMessage()); + extensionEnabled = false; + } + if (extensionEnabled) { + System.out.println("BSim extension enabled"); + } + else { + forceShutdown = true; // Force a shutdown, because extension isn't enabled + stopCommand(); + } + } + + /** + * Stop the running PostgreSQL processes on the local host. No authentication is required to shutdown + * the server. User must be the process owner. + * @throws IOException if postgres cannot be discovered or stopped + * @throws InterruptedException if the stop command is interrupted + */ + private void stopCommand() throws IOException, InterruptedException { + discoverPostgresInstall(); + List command = new ArrayList(); + command.add(postgresControl.getAbsolutePath()); + command.add("stop"); + command.add("-D"); + command.add(dataDirectory.getAbsolutePath()); + if (forceShutdown) { + command.add("-m"); + command.add("fast"); // Does not wait for clients to disconnect, all active transactions rolled back + } + int res = runCommand(null, command, loadLibraryVar, loadLibraryValue); + if (res != 0) { + throw new IOException("Error shutting down postgres server process"); + } + System.out.println("Server shutdown complete"); + } + + /** + * Trigger a server running on the local host to rescan its identity file to pickup + * any changes to the user mapping + * @throws IOException if creating a new user fails + * @throws InterruptedException if the reload command is interrupted + */ + private void reloadIdent() throws IOException, InterruptedException { + List command = new ArrayList(); + command.add(postgresControl.getAbsolutePath()); + command.add("reload"); + command.add("-D"); + command.add(dataDirectory.getAbsolutePath()); + command.add("-s"); + int res = runCommand(null, command, loadLibraryVar, loadLibraryValue); + if (res != 0) { + throw new IOException("Error creating new user"); + } + } + + /** + * Update the PostgreSQL identity map (pg_ident.conf) adding a map from + * the currently active -commonName- to -username- + * @param username the user name to add + * @throws IOException if the postgres ident file is invalid + */ + private void addCertificateName(String username) throws IOException { + File identFile = new File(dataDirectory, POSTGRES_IDENTFILE); + File copyFile = new File(dataDirectory, POSTGRES_IDENTFILE + ".copy"); + if (!identFile.isFile()) { + throw new IOException("Missing ident file: " + identFile.getAbsolutePath()); + } + ServerConfig.patchIdent(identFile, copyFile, POSTGRES_MAP_IDENTIFIER, commonName, username, + true); + FileUtilities.copyFile(copyFile, identFile, false, null); + } + + /** + * Returns a list of all users registered with the BSim server. + * + * Note: This will return all users minus those created by Postgres (those that + * start with 'pg_'. + * + * @param dataDirectory the location of the Postgres database files + * @return map of database users and their admin status + * @throws Exception if there's a problem initializing the Application or searching for Postgres + */ + public Map getUserRolesCommand(String dataDirectory) throws Exception { + + String[] params = { dataDirectory }; + scanDataDirectory(params, 0); + initializeApplication(); + discoverPostgresInstall(); + + if (connectingUserName == null) { + connectingUserName = ClientUtil.getUserName(); + } + + adminPasswordData = null; + + localConnection = getOrCreateLocalConnection(); + + StringBuilder buffer = new StringBuilder(); + buffer.append("SELECT rolname, rolsuper from pg_roles"); + try (Statement st = localConnection.createStatement()) { + Map userToAdminMap = new HashMap<>(); + try (ResultSet rs = st.executeQuery(buffer.toString())) { + while (rs.next()) { + String user = rs.getString(1); + + if (user.startsWith("pg_")) { // default postgres role - ignore + continue; + } + Boolean isAdmin = rs.getBoolean(2); + userToAdminMap.put(user, isAdmin); + } + return userToAdminMap; + } + } + catch (SQLException e) { + Msg.error(this, "Error retrieving user roles from the Postgres database", e); + } + finally { + localConnection.close(); + } + + return Collections.emptyMap(); + } + + /** + * Add a new user to the currently running server on the local host. + * A connection is established, using the local interface, and the "CREATE ROLE" command + * is executed. If the server is configured to require certificate authentication on + * remote connections, the user must have provided a distinguished name associated with + * the certificate, which is then mapped to the new username. + * @throws GeneralSecurityException if using PKI and no Distinguished Name is found + * @throws Exception if there's a problem initializing the Application of discovering the Postgres installation + */ + private void addUserCommand() throws GeneralSecurityException, Exception { + discoverPostgresInstall(); + initializeDataDirectory(); // Needed to pick up authentication settings + if (hostAuthentication == AUTHENTICATION_PKI) { + if (distinguishedName == null || commonName == null) { + throw new GeneralSecurityException("Distinguished name required (dn=\"..\")"); + } + } + StringBuilder resultMessage = new StringBuilder(); + resultMessage.append("Added user: "); + resultMessage.append(specifiedUserName); + boolean resetPassword = (hostAuthentication == AUTHENTICATION_PASSWORD); + + adminPasswordData = null; + + localConnection = getOrCreateLocalConnection(); + + StringBuilder buffer = new StringBuilder(); + buffer.append("CREATE ROLE \""); + buffer.append(specifiedUserName); + buffer.append("\" WITH LOGIN"); + + try (Statement st = localConnection.createStatement()) { + st.executeUpdate(buffer.toString()); + } + catch (SQLException err) { + if (!err.getMessage().contains("already exists")) { // Suppress already exists error message + throw err; + } + resultMessage.append(" (already present)"); // Record that user is already added + resetPassword = false; + } + finally { + if (resetPassword) { + resetPassword(localConnection, specifiedUserName); + } + localConnection.close(); + } + + if (hostAuthentication == AUTHENTICATION_PKI) { + addCertificateName(specifiedUserName); + reloadIdent(); + System.out.println("Linking distinguished name to user: " + specifiedUserName); + return; + } + System.out.println(resultMessage.toString()); + } + + /** + * Returns a connection to a local Postgres database. If a connection has not yet + * been established, it creates one. + * + * @return the database connection + * @throws Exception if there's an error creating the connection + */ + private Connection getOrCreateLocalConnection() throws Exception { + + try { + if (localConnection == null || localConnection.isClosed()) { + localConnection = createLocalConnection(); + } + } + catch (SQLException | IOException e) { + Msg.error(this, "Error creating connection to Postgres database", e); + throw e; + } + return localConnection; + } + + /** + * On a server running on the local host, remove the specified username. + * A local connection is created and the "DROP ROLE" command is run. If + * the server uses PKI authentication, the PostgreSQL identity file is + * scanned, and any mapping that matches dropped name is also removed. + * @throws Exception + */ + private void dropUserCommand() throws Exception { + discoverPostgresInstall(); + initializeDataDirectory(); + boolean userDoesNotExist = false; + localConnection = getOrCreateLocalConnection(); + StringBuilder buffer = new StringBuilder(); + buffer.append("DROP ROLE \""); + buffer.append(specifiedUserName); + buffer.append('\"'); + + try (Statement st = localConnection.createStatement()) { + st.executeUpdate(buffer.toString()); + } + catch (SQLException err) { + if (!err.getMessage().contains("does not exist")) { + throw err; + } + userDoesNotExist = true; + } + finally { + localConnection.close(); + } + + if (hostAuthentication == AUTHENTICATION_PKI || localAuthentication == AUTHENTICATION_PKI) { + File identFile = new File(dataDirectory, POSTGRES_IDENTFILE); + File copyFile = new File(dataDirectory, POSTGRES_IDENTFILE + ".copy"); + if (!identFile.isFile()) { + throw new IOException("Missing ident file: " + identFile.getAbsolutePath()); + } + ServerConfig.patchIdent(identFile, copyFile, POSTGRES_MAP_IDENTIFIER, commonName, + specifiedUserName, false); + FileUtilities.copyFile(copyFile, identFile, false, null); + reloadIdent(); + } + if (userDoesNotExist) { + System.out.println("User " + specifiedUserName + " does not exist"); + } + else { + System.out.println("Removed user: " + specifiedUserName); + } + } + + /** + * The data directory for a server (which must not be running) is reconfigured + * with new local and remote authentication options, and the port may be reconfigured as well. + * Database records are unaltered. + * The user submits "auth=..", "localAuth=..", and "port=.." options on the command line: + * among those submitted, any option that doesn't match the current configuration is changed. + * @throws IOException if the postgres installation cannot be found + * @throws InterruptedException if the postgres installation cannot be found + * @throws SAXException if the {@link #tuneConfig(File, File, File, File, File)} call fails + * @throws GeneralSecurityException if there is no Distinguished Name supplied + */ + private void resetCommand() + throws IOException, InterruptedException, SAXException, GeneralSecurityException { + discoverPostgresInstall(); + File configFile = new File(dataDirectory, POSTGRES_CONFIGFILE); + File hbaFile = new File(dataDirectory, POSTGRES_CONNECTFILE); + if (!configFile.exists()) { + throw new IOException("Data directory not initialized: run \"bsim_ctl start\" first"); + } + int requestedLocalAuth = localAuthentication; + int requestedHostAuth = hostAuthentication; + int requestedPort = port; + port = -1; + recoverConfigurationParameters(configFile, hbaFile); + if (isServerRunning()) { + throw new IOException("Cannot modify settings on running server"); + } + if ((!authConfigPresent || (requestedHostAuth == hostAuthentication && + requestedLocalAuth == localAuthentication)) && + (requestedPort == -1 || requestedPort == port)) { + System.out.println("No changes to make"); + return; + } + + File serverConfigFile = Application.getModuleDataFile("serverconfig.xml").getFile(false); + File configCopy = new File(dataDirectory, POSTGRES_CONFIGFILE + ".orig"); + if (!configCopy.exists()) { + throw new IOException( + "Original configuration file not present: " + configCopy.getAbsolutePath()); + } + File hbaCopy = new File(dataDirectory, POSTGRES_CONNECTFILE + ".orig"); + if (!hbaCopy.exists()) { + throw new IOException( + "Original connection file not present: " + hbaCopy.getAbsolutePath()); + } + if (requestedPort != -1 && requestedPort != port) { + port = requestedPort; + System.out.println("Server will now listen on port " + Integer.toString(port)); + } + boolean newRemotePki = false; // Are we newly enabling remote pki + boolean newLocalPki = false; // Are we newly enabling local pki + + if (authConfigPresent && requestedLocalAuth != localAuthentication) { + localAuthentication = requestedLocalAuth; + System.out.print("New local authentication: "); + if (localAuthentication == AUTHENTICATION_PASSWORD) { + System.out.println("password"); + } + else if (localAuthentication == AUTHENTICATION_PKI) { + System.out.println("pki"); + newLocalPki = true; + if (commonName == null) { + throw new GeneralSecurityException("Distinguished name required (dn=\"..\")"); + } + } + else if (localAuthentication == AUTHENTICATION_NONE) { + System.out.println("none"); + } + } + if (authConfigPresent && requestedHostAuth != hostAuthentication) { + hostAuthentication = requestedHostAuth; + System.out.print("New host authentication: "); + if (hostAuthentication == AUTHENTICATION_NONE) { + System.out.println("none"); + } + else if (hostAuthentication == AUTHENTICATION_PASSWORD) { + System.out.println("password"); + } + else if (hostAuthentication == AUTHENTICATION_PKI) { + System.out.println("pki"); + newRemotePki = true; + } + else { + System.out.println("unknown"); + } + } + if (newLocalPki || newRemotePki) { + checkCertAuthorityFile(); + File rootCA = new File(dataDirectory, POSTGRES_ROOTCA); + FileUtilities.copyFile(certAuthorityFile, rootCA, false, null); + } + if (newLocalPki) { + addCertificateName(connectingUserName); + } + + tuneConfig(configCopy, configFile, hbaCopy, hbaFile, serverConfigFile); + } + + /** + * Reset the password for -username- to DEFAULT_PASSWORD + * @param pdb is the connection over which to issue the command + * @param username is the user name to reset + * @throws SQLException if the sql query fails + */ + private void resetPassword(Connection pdb, String username) throws SQLException { + StringBuilder buffer = new StringBuilder(); + buffer.append("ALTER ROLE \""); + buffer.append(username); + buffer.append("\" WITH PASSWORD '"); + buffer.append(DEFAULT_PASSWORD); + buffer.append('\''); + executeSQLStatement(pdb, buffer.toString()); + } + + /** + * Reset the PostgreSQL password associated with the specified user name on the local server. + * The user submitting this request on behalf of the specified user may need to + * enter their own password or passphrase to authenticate with the server. + * @throws Exception if there's a problem getting a connection to the Postgres database + */ + private void passwordCommand() throws Exception { + localConnection = getOrCreateLocalConnection(); + System.out.println("Resetting password for user: " + specifiedUserName); + + try { + resetPassword(localConnection, specifiedUserName); + System.out.println("Password reset complete"); + } + finally { + localConnection.close(); + } + } + + private void privilegeCommand() throws Exception { + localConnection = getOrCreateLocalConnection(); + try { + if (adminPrivilegeRequested) { + System.out.println("Granting admin privileges to " + specifiedUserName); + executeSQLStatement(localConnection, + "ALTER ROLE " + specifiedUserName + " SUPERUSER CREATEROLE CREATEDB"); + } + else { + System.out.println("Revoking admin privileges from " + specifiedUserName); + executeSQLStatement(localConnection, + "ALTER ROLE " + specifiedUserName + " NOSUPERUSER NOCREATEROLE NOCREATEDB"); + } + } + finally { + localConnection.close(); + } + } + + /** + * Parse the command-line. First argument is always a command, which may + * require additional arguments. Additional optional arguments may follow + * @param params is the array of command-line arguments + * @throws ArgumentException if the data directory cannot be scanned or the authentication method is invalid + * @throws IOException if the data directory cannot be scanned + */ + private void readCommandLine(String[] params) throws ArgumentException, IOException { + String command = params[0]; + int slot = 1; + if (command.equals(START_COMMAND)) { + scanDataDirectory(params, slot); + slot += 1; + } + else if (command.equals(STOP_COMMAND)) { + scanDataDirectory(params, slot); + slot += 1; + } + else if (command.equals(ADDUSER_COMMAND)) { + scanDataDirectory(params, slot); + slot += 1; + scanUsername(params, slot); + slot += 1; + } + else if (command.equals(DROPUSER_COMMAND)) { + scanDataDirectory(params, slot); + slot += 1; + scanUsername(params, slot); + slot += 1; + } + else if (command.equals(PASSWORD_COMMAND)) { + scanUsername(params, slot); + slot += 1; + } + else if (command.equals(RESET_COMMAND)) { + scanDataDirectory(params, slot); + slot += 1; + } + else if (command.equals(PRIVILEGE_COMMAND)) { + scanUsername(params, slot); + slot += 1; + scanPrivilege(params, slot); + slot += 1; + } + else { + throw new ArgumentException("Unknown command: " + command); + } + + // Scan for optional arguments + boolean sawNoLocalAuth = false; + for (int i = slot; i < params.length; ++i) { + String option = params[i]; + if (option.startsWith(PORT_OPTION)) { + port = Integer.parseInt(option.substring(PORT_OPTION.length())); + } + else if (option.startsWith(CAFILE_OPTION)) { + certAuthorityFile = new File(option.substring(CAFILE_OPTION.length())); + } + else if (option.startsWith(AUTH_OPTION)) { + authConfigPresent = true; + String type = option.substring(AUTH_OPTION.length()); + if (type.equals("pki")) { + hostAuthentication = AUTHENTICATION_PKI; + localAuthentication = AUTHENTICATION_PKI; + } + else if (type.equals("password")) { + hostAuthentication = AUTHENTICATION_PASSWORD; + localAuthentication = AUTHENTICATION_PASSWORD; + } + else if (type.equals("trust") || type.equals("none")) { + hostAuthentication = AUTHENTICATION_NONE; + localAuthentication = AUTHENTICATION_NONE; + } + else { + throw new ArgumentException( + "Unknown authentication method: " + type + " : options are trust, pki"); + } + } + else if (option.startsWith(DN_OPTION)) { + distinguishedName = option.substring(DN_OPTION.length()); + validateDistinguishedName(); + } + else if (option.startsWith(CERT_OPTION)) { + certParameter = option.substring(CERT_OPTION.length()); + } + else if (option.startsWith(NO_LOCAL_AUTH_OPTION)) { + sawNoLocalAuth = true; + } + else if (option.equals(FORCE_OPTION)) { + forceShutdown = true; + } + else if (option.startsWith(USER_OPTION)) { + connectingUserName = option.substring(USER_OPTION.length()); + } + else { + throw new ArgumentException("Unknown option: " + option); + } + } + if (sawNoLocalAuth) { // Turn off authentication for local connections + authConfigPresent = true; + localAuthentication = AUTHENTICATION_NONE; + } + if (connectingUserName == null) { + connectingUserName = ClientUtil.getUserName(); + } + } + + /** + * Runs the command specified by the given set of params. + * + * @param params the command parameters + * @param monitor the task monitor + * @throws Exception if there is a problem executing the command + */ + public void run(String[] params, TaskMonitor monitor) throws Exception { + String command = params[0]; + try { + readCommandLine(params); + initializeApplication(); + if (command.equals(START_COMMAND)) { + startCommand(); + } + else if (command.equals(STOP_COMMAND)) { + stopCommand(); + } + else if (command.equals(ADDUSER_COMMAND)) { + addUserCommand(); + } + else if (command.equals(DROPUSER_COMMAND)) { + dropUserCommand(); + } + else if (command.equals(RESET_COMMAND)) { + resetCommand(); + } + else if (command.equals(PASSWORD_COMMAND)) { + passwordCommand(); + } + else if (command.equals(PRIVILEGE_COMMAND)) { + privilegeCommand(); + } + } + catch (SAXException e1) { + System.err.println("Error in server configuation data"); + System.err.println(e1.getMessage()); + throw e1; + } + catch (IOException e1) { + System.err.println("Error configuring PostgreSQL for BSim"); + System.err.println(e1.getMessage()); + throw e1; + } + catch (InterruptedException e) { + System.err.println("Command was interrupted"); + System.err.println(e.getMessage()); + throw e; + } + catch (SQLException e) { + System.err.println("Error connecting to the database"); + System.err.println(e.getMessage()); + throw e; + } + catch (GeneralSecurityException e) { + System.err.println("Error establishing server certificate"); + System.err.println(e.getMessage()); + throw e; + } + catch (ArgumentException e) { + System.err.println("Error in command line arguments"); + System.err.println(e.getMessage()); + throw e; + } + catch (ClassNotFoundException e) { + System.err.println("Could not find PostgreSQL JDBC driver"); + System.err.println(e.getMessage()); + throw e; + } + try { + cleanupPasswordData(); + } + catch (IOException e) { + e.printStackTrace(); + } + } + + /** + * Runs the command specified by the given set of params. + * + * @param params the command parameters + * @throws Exception if there is a problem executing the command + */ + public void run(String[] params) throws Exception { + run(params, TaskMonitor.DUMMY); + } + + @Override + public void launch(GhidraApplicationLayout ghidraLayout, String[] params) { + if (params.length <= 1) { + //@formatter:off + System.err.println("USAGE:"); + System.err.println(" bsim_ctl start [auth=pki|password|trust] [--noLocalAuth] [cafile=] [dn=\"..\"]"); + System.err.println(" stop [--force]"); + System.err.println(" adduser [dn=\"..\"]"); + System.err.println(" dropuser "); + System.err.println(" changeauth [auth=pki|password|trust] [--noLocalAuth] [cafile=]"); + System.err.println(" resetpassword "); + System.err.println(" changeprivilege admin|user"); + System.err.println(); + System.err.println("Global options:"); + System.err.println(" port="); + System.err.println(" user="); + System.err.println(" cert="); + System.err.println(); + //@formatter:on + return; + } + layout = ghidraLayout; // Save layout for when we need to initialize application + try { + run(params); + } + catch (RuntimeException e) { + e.printStackTrace(); + System.exit(1); + } + catch (Exception e) { + System.exit(1); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimDBConnectTaskCoordinator.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimDBConnectTaskCoordinator.java new file mode 100644 index 0000000000..669de234c7 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimDBConnectTaskCoordinator.java @@ -0,0 +1,162 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.sql.Connection; +import java.sql.SQLException; + +import ghidra.features.bsim.query.client.CancelledSQLException; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.*; + +/** + * Provides the ability to synchronize concurrent connection task + * instances within the same thread. This can occur within the swing thread due to the presence + * of a modal task dialog event queue. It also allows password cancelation to be propogated to the + * other tasks(s). + */ +public class BSimDBConnectTaskCoordinator { + + private final BSimServerInfo serverInfo; + + private Exception exc = null; + private boolean isCancelled = false; + private int count = 0; + + public BSimDBConnectTaskCoordinator(BSimServerInfo serverInfo) { + this.serverInfo = serverInfo; + } + + private void clear() { + exc = null; + isCancelled = false; + count = 0; + } + + /** + * Initiate a DB connection. + * @param connectionSupplier DB connection supplier + * @return DB connection + * @throws SQLException if a database connection error occured + * @throws CancelledSQLException if task was cancelled (password entry cancelled) + */ + public Connection getConnection(DBConnectionSupplier connectionSupplier) throws SQLException { + + // Use task to establish initial connection + DBConnectTask connectTask = new DBConnectTask(connectionSupplier); + try { + //@formatter:off + TaskBuilder.withTask(connectTask) + .setTitle("BSim DB Connection...") + .setCanCancel(false) + .setHasProgress(false) + .launchModal(); + //@formatter:on + + synchronized (BSimDBConnectTaskCoordinator.this) { + Connection c = connectTask.getConnection(); + if (c != null) { + return c; + } + + if (isCancelled) { + throw new CancelledSQLException("Password entry was cancelled"); + } + if (exc instanceof SQLException e) { + throw e; + } + if (exc instanceof RuntimeException e) { + throw e; + } + throw new RuntimeException(exc); + } + } + finally { + synchronized (BSimDBConnectTaskCoordinator.this) { + if (--count == 0) { + clear(); + } + } + } + } + + /** + * DB connection supplier + */ + public interface DBConnectionSupplier { + + /** + * Get a database connection. + * @return database connection + * @throws CancelledException if connection attempt cancelled + * @throws SQLException if a database connection error occurs + */ + public Connection get() throws CancelledException, SQLException; + } + + /** + * Task for connecting to Postgres DB server with Swing thread. + */ + private class DBConnectTask extends Task { + + private Connection c; + private DBConnectionSupplier connectionSupplier; + + /** + * Server Connect Task constructor + * @param connectionSupplier DB connection supplier + */ + DBConnectTask(DBConnectionSupplier connectionSupplier) { + super("BSim Connecting to " + serverInfo, false, false, true); + this.connectionSupplier = connectionSupplier; + } + + Connection getConnection() { + return c; + } + + /** + * Completes and necessary authentication and obtains a DB connection. + * If a connection error occurs, an exception will be stored. + * @throws CancelledException if task cancelled + * @see ghidra.util.task.Task#run(ghidra.util.task.TaskMonitor) + */ + @Override + public void run(TaskMonitor monitor) throws CancelledException { + synchronized (BSimDBConnectTaskCoordinator.this) { + monitor.setMessage("Connecting..."); + ++count; + if (isCancelled) { + throw new CancelledException(); + } + if (exc != null) { + return; + } + try { + c = connectionSupplier.get(); + } + catch (CancelledException e) { + isCancelled = true; + throw e; + } + catch (Exception e) { + exc = e; + } + } + } + + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimInitializer.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimInitializer.java new file mode 100644 index 0000000000..64e419cdb8 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimInitializer.java @@ -0,0 +1,34 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import ghidra.features.bsim.query.postgresql.Handler; +import ghidra.framework.ModuleInitializer; + +public class BSimInitializer implements ModuleInitializer { + + @Override + public void run() { + // Register the "postgresql" protocol, so users can use "postgresql://hostname/repo" URLs + Handler.registerHandler(); + } + + @Override + public String getName() { + return "BSim Module"; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimJDBCDataSource.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimJDBCDataSource.java new file mode 100644 index 0000000000..c0ca147904 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimJDBCDataSource.java @@ -0,0 +1,51 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.sql.Connection; +import java.sql.SQLException; + +import ghidra.features.bsim.query.FunctionDatabase.ConnectionType; +import ghidra.features.bsim.query.FunctionDatabase.Status; + +public interface BSimJDBCDataSource { + + Status getStatus(); + + /** + * Get DB {@link Connection} object performing any required authentication. + * @return {@link Connection} object + * @throws SQLException if connection fails + */ + Connection getConnection() throws SQLException; + + ConnectionType getConnectionType(); + + /** + * Get the server info that corresponds to this data source. It is important to note + * that the returned instance is normalized for the purpose of caching and may not + * match the original server info object used to obtain this data source instance. + * @return server info + */ + BSimServerInfo getServerInfo(); + + /** + * Get the number of active connections in the associated connection pool + * @return number of active connections + */ + int getActiveConnections(); + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimPostgresDBConnectionManager.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimPostgresDBConnectionManager.java new file mode 100644 index 0000000000..c0dafe4370 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimPostgresDBConnectionManager.java @@ -0,0 +1,339 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.net.URL; +import java.sql.Connection; +import java.sql.SQLException; +import java.util.HashMap; + +import javax.security.auth.callback.NameCallback; +import javax.security.auth.callback.PasswordCallback; + +import org.apache.commons.dbcp2.BasicDataSource; +import org.apache.commons.lang3.StringUtils; + +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.FunctionDatabase.ConnectionType; +import ghidra.features.bsim.query.FunctionDatabase.Status; +import ghidra.framework.client.*; +import ghidra.util.Msg; +import ghidra.util.exception.CancelledException; + +public class BSimPostgresDBConnectionManager { + + private static final String DRIVER_CLASS_NAME = "org.postgresql.Driver"; + private static final int CONN_POOL_SIZE = 2; + private static final int CONN_POOL_MAX_IDLE = 2; + + private static HashMap dataSourceMap = new HashMap<>(); + + public static BSimPostgresDataSource getDataSource(BSimServerInfo postgresServerInfo) { + if (postgresServerInfo.getDBType() != DBType.postgres) { + throw new IllegalArgumentException("expected postgres server info"); + } + return dataSourceMap.computeIfAbsent(postgresServerInfo, + info -> new BSimPostgresDataSource(info)); + } + + @Deprecated + public static BSimPostgresDataSource getDataSource(URL postgresUrl) { + return getDataSource(new BSimServerInfo(postgresUrl)); + } + + public static BSimPostgresDataSource getDataSourceIfExists(BSimServerInfo serverInfo) { + return dataSourceMap.get(serverInfo); + } + + private static synchronized void remove(BSimServerInfo serverInfo) { + BSimPostgresDataSource ds = dataSourceMap.get(serverInfo); + if (ds == null) { + return; + } + int n = ds.bds.getNumActive(); + if (n != 0) { + System.out + .println("Unable to remove data source which has " + n + " active connections"); + return; + } + ds.close(); + dataSourceMap.remove(serverInfo); + } + + public static class BSimPostgresDataSource implements BSimJDBCDataSource { // NOTE: can be renamed + + private final BSimServerInfo serverInfo; + + private ConnectionType connectionType = ConnectionType.SSL_No_Authentication; + private boolean successfulConnection = false; + + private BasicDataSource bds = new BasicDataSource(); + private BSimDBConnectTaskCoordinator taskCoordinator; + + private BSimPostgresDataSource(BSimServerInfo serverInfo) { + this.serverInfo = serverInfo; + this.taskCoordinator = new BSimDBConnectTaskCoordinator(serverInfo); + } + + @Override + public BSimServerInfo getServerInfo() { + return serverInfo; + } + + public void initializeFrom(BSimPostgresDataSource otherDs) { + if (!otherDs.successfulConnection || + otherDs.connectionType != ConnectionType.SSL_Password_Authentication) { + return; + } + setDefaultProperties(); + setSSLProperties(); + bds.setUsername(otherDs.getUserName()); + bds.setPassword(otherDs.bds.getPassword()); + successfulConnection = true; + } + + public String getUserName() { + return bds.getUsername(); + } + + public void setPreferredUserName(String userName) { + bds.setUsername(userName); + } + + public void dispose() { + remove(serverInfo); + } + + private void close() { + try { + bds.close(); + } + catch (SQLException e) { + // ignore + } + bds.setPassword(null); + } + + @Override + public Status getStatus() { + if (bds.isClosed()) { + return Status.Unconnected; + } + if (successfulConnection) { + return Status.Ready; + } + return Status.Error; + } + + @Override + public int getActiveConnections() { + return bds.getNumActive(); + } + + /** + * Update password on {@link BasicDataSource} for use with future connect attempts. + * Has no affect if username does not match username on data source. + * @param username username + * @param newPassword updated password + */ + public void setPassword(String username, char[] newPassword) { + if (username.equals(bds.getUsername())) { + bds.setPassword(String.valueOf(newPassword)); + } + } + + private void setDefaultProperties() { + //Set database driver name + bds.setDriverClassName(DRIVER_CLASS_NAME); + + //Set database url + int port = serverInfo.getPort(); + bds.setUrl("jdbc:postgresql://" + serverInfo.getServerName() + + (port > 0 ? (":" + Integer.toString(port)) : "") + "/" + serverInfo.getDBName()); + + //Set the connection pool size + bds.setInitialSize(CONN_POOL_SIZE); + + // Set maximum number of idle connections + bds.setMaxIdle(CONN_POOL_MAX_IDLE); + + // Validate connection borrowed from pool + bds.setValidationQuery("SELECT 1"); + bds.setTestOnBorrow(true); + + // bds.setLogAbandoned(true); + // bds.setAbandonedUsageTracking(true); + + // PGStatement.setPrepareThreshold(2) + bds.addConnectionProperty("prepareThreshold", "2"); + } + + private void setSSLProperties() { + bds.addConnectionProperty("sslmode", "require"); + bds.addConnectionProperty("sslfactory", "ghidra.net.ApplicationSSLSocketFactory"); + } + + @Override + public synchronized Connection getConnection() throws SQLException { + + if (successfulConnection) { + try { + // attempt to reuse pooled connection or current settings + try { + return bds.getConnection(); + } + catch (SQLException e) { + // ignore + } + + // Give one restart attempt (possible password change) + bds.restart(); + return bds.getConnection(); + } + catch (SQLException e) { + successfulConnection = false; + bds.close(); + BasicDataSource newBds = new BasicDataSource(); + newBds.setUsername(bds.getUsername()); + bds.setPassword(null); // sanitize old instance + bds = newBds; + // fall-through for clean start + } + finally { + Msg.debug(this, serverInfo + " getConnection: active=" + bds.getNumActive() + + " idle=" + bds.getNumIdle()); + } + } + + setDefaultProperties(); + + return taskCoordinator.getConnection(() -> connect()); + } + + @Override + public ConnectionType getConnectionType() { + return connectionType; + } + + @Override + public boolean equals(Object obj) { + if (obj instanceof BSimPostgresDataSource ds) { + return bds.getUrl().equals(ds.bds.getUrl()); + } + return false; + } + + @Override + public int hashCode() { + return bds.getUrl().hashCode(); + } + + /** + * Establish Postgres DB {@link Connection} performing any required authentication. + * @throws SQLException if connection or authentication error occurs + * @throws CancelledException if connection cancelled by user + */ + private Connection connect() throws SQLException, CancelledException { + + String userName = bds.getUsername(); + bds.setUsername(StringUtils.isBlank(userName) ? ClientUtil.getUserName() : userName); + bds.setPassword(null); + connectionType = ConnectionType.SSL_No_Authentication; + try { + // Specify SSL connection properties + setSSLProperties(); + Connection c = bds.getConnection(); + successfulConnection = true; + return c; + } + catch (SQLException e) { + // TODO: Need to verify these + if (e.getMessage().contains("password-based authentication") || + e.getMessage().contains("SCRAM-based") || + e.getMessage().contains("password authentication failed")) { + // Use Ghidra's authentication infrastructure + connectionType = ConnectionType.SSL_Password_Authentication; // Try again with a password + // fallthru to second attempt at getConnection + } + else if (e.getMessage().contains("SSL on") && + e.getMessage().contains("no pg_hba.conf entry")) { + connectionType = ConnectionType.Unencrypted_No_Authentication; // Try again without any SSL + bds.removeConnectionProperty("sslmode"); + bds.removeConnectionProperty("sslfactory"); + // fallthru to second attempt at getConnection + } + else { + throw e; + } + } + finally { + Msg.debug(this, serverInfo + " getConnection: active=" + bds.getNumActive() + + " idle=" + bds.getNumIdle()); + } + + String loginError = null; + while (true) { + ClientAuthenticator clientAuthenticator = null; + if (connectionType == ConnectionType.SSL_Password_Authentication) { + clientAuthenticator = ClientUtil.getClientAuthenticator(); + if (clientAuthenticator == null) { // Make sure authenticator is registered + throw new SQLException("No registered authenticator"); + } + NameCallback nameCb = new NameCallback("User ID:"); + nameCb.setName(bds.getUsername()); + PasswordCallback passCb = new PasswordCallback("Password:", false); + try { + if (!clientAuthenticator.processPasswordCallbacks( + "BSim Database Authentication", "BSim Database Server", + serverInfo.toString(), nameCb, passCb, null, null, loginError)) { + throw new CancelledException(); + } + bds.setPassword(new String(passCb.getPassword())); + // User may have specified new username, or this may return NULL + userName = nameCb.getName(); + if (!StringUtils.isBlank(userName)) { + bds.setUsername(userName); + } + } + finally { + passCb.clearPassword(); + } + } + try { + Connection c = bds.getConnection(); + successfulConnection = true; + return c; + } + catch (SQLException e) { + if ((clientAuthenticator instanceof DefaultClientAuthenticator) && + e.getMessage().contains("password authentication failed")) { + // wrong password provided via popup dialog - try again + loginError = "Access denied: " + serverInfo; + continue; + } + connectionType = ConnectionType.SSL_No_Authentication; + throw e; + } + finally { + Msg.debug(this, serverInfo + " getConnection: active=" + bds.getNumActive() + + " idle=" + bds.getNumIdle()); + } + } + } + + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimServerInfo.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimServerInfo.java new file mode 100644 index 0000000000..2fe9cfbc74 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BSimServerInfo.java @@ -0,0 +1,326 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.io.Closeable; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.Objects; + +import org.apache.commons.lang3.StringUtils; + +public class BSimServerInfo implements Comparable { + + /** + * Default port used for {@link DBType#postgres} server + */ + public static final int DEFAULT_POSTGRES_PORT = 5432; + + /** + * Default port used for {@link DBType#elastic} server + */ + public static final int DEFAULT_ELASTIC_PORT = 9200; + + /** + * File extension imposed for {@link DBType#file} server. + * This is a rigid H2 database convention. + */ + public static final String H2_FILE_EXTENSION = ".mv.db"; + + /** + * Enumerated Database Types + */ + public enum DBType { + postgres, elastic, file; + } + + private final DBType dbType; + private final String host; + private final int port; + private final String dbName; + + private String shortDbName; // lazy: short DB Name + + /** + * Construct a new {@link BSimServerInfo} object + * @param dbType BSim DB type + * @param host host name (ignored for {@link DBType#file}) + * @param port port number (ignored for {@link DBType#file}) + * @param dbName name of database (simple database name except for {@link DBType#file} + * which should reflect an absolute file path. On Windows OS the path may start with a + * drive letter. + * @throws IllegalArgumentException if invalid arguments are specified + */ + public BSimServerInfo(DBType dbType, String host, int port, String dbName) { + Objects.requireNonNull(dbType, "DBType must be specified"); + this.dbType = dbType; + + if ((dbType == DBType.postgres || dbType == DBType.elastic) && StringUtils.isEmpty(host)) { + throw new IllegalArgumentException("host required"); + } + this.host = host; + + if (port <= 0) { + port = -1; + } + if (dbType == DBType.postgres && port <= 0) { + port = DEFAULT_POSTGRES_PORT; + } + if (dbType == DBType.elastic && port <= 0) { + port = DEFAULT_ELASTIC_PORT; + } + this.port = port; + + dbName = dbName.trim(); + if (StringUtils.isEmpty(dbName)) { + throw new IllegalArgumentException("Non-empty dbName required"); + } + if (dbType == DBType.file) { + // transform dbName into acceptable H2 DB file path + dbName = dbName.replace("\\", "/"); + if ((!dbName.startsWith("/") && !isWindowsFilePath(dbName)) || dbName.endsWith("/")) { + throw new IllegalArgumentException("Invalid absolute file path: " + dbName); + } + if (!dbName.endsWith(H2_FILE_EXTENSION)) { + dbName += H2_FILE_EXTENSION; + } + } + else if (dbName.contains("/") || dbName.contains("\\")) { // may want additional validation + throw new IllegalArgumentException("Invalid " + dbType + " dbName: " + dbName); + } + this.dbName = dbName; + } + + /** + * Construct a new {@link BSimServerInfo} object from a suitable database URL + * (i.e., {@code postgresql:}, {@code https:}, {@code elastic:}, {@code file:}). + * @param url supported BSim database URL + * @throws IllegalArgumentException if unsupported URL protocol specified + */ + public BSimServerInfo(URL url) throws IllegalArgumentException { + + DBType t = null; + String path = url.getPath(); + String protocol = url.getProtocol(); + if (protocol.equals("postgresql")) { + t = DBType.postgres; + host = checkURLField(url.getHost(), "host"); + int p = url.getPort(); + port = p <= 0 ? DEFAULT_POSTGRES_PORT : p; + } + else if (protocol.equals("https") || protocol.equals("elastic")) { + t = DBType.elastic; + host = checkURLField(url.getHost(), "host"); + int p = url.getPort(); + port = p <= 0 ? DEFAULT_ELASTIC_PORT : p; + } + else if (protocol.startsWith("file")) { + t = DBType.file; + host = null; + port = -1; + if (!"".equals(url.getHost())) { + throw new IllegalArgumentException("Remote file URL not supported: " + url); + } + } + else { + throw new IllegalArgumentException("Unsupported BSim URL protocol: " + protocol); + } + dbType = t; + + if (dbType == DBType.postgres || dbType == DBType.elastic) { + if (!path.startsWith("/")) { + throw new IllegalArgumentException("Missing dbName in URL: " + url); + } + path = path.substring(1).strip(); + } + path = checkURLField(path, "path"); + if (dbType == DBType.file) { + if (path.endsWith("/")) { + throw new IllegalArgumentException("Missing DB filepath in URL: " + url); + } + if (!path.endsWith(H2_FILE_EXTENSION)) { + path += H2_FILE_EXTENSION; + } + // TODO: handle Windows path with drive letter - need to remove leading '/' + } + else if (path.contains("/")) { + throw new IllegalArgumentException("Invalid dbName in URL: " + path); + } + dbName = path; + } + + private static String checkURLField(String val, String name) { + if (StringUtils.isEmpty(val)) { + throw new IllegalArgumentException("Invalid " + name + " in URL"); + } + return val.trim(); + } + + /** + * Determine if this server info corresponds to Windows OS file path. + * @return true if this server info corresponds to Windows OS file path. + */ + public boolean isWindowsFilePath() { + return dbType == DBType.file && isWindowsFilePath(dbName); + } + + /** + * Check for Windows path after all '/' chars have been converted to '\' chars. + * Example: {@code C:/a/b/c} + * @param path absolute file path + * @return true if path appears to be windows path with a drive letter + */ + private static boolean isWindowsFilePath(String path) { + if (path.length() < 4) { + return false; + } + if (!Character.isLetter(path.charAt(0)) || path.charAt(1) != ':') { + return false; + } + char c = path.charAt(2); + if (c != '/') { + return false; + } + c = path.charAt(3); + return c != '/'; + } + + /** + * Return BSim server info in URL format + * @return BSim server info in URL format + */ + public String toURLString() { + switch (dbType) { + case postgres: + return "postgresql://" + host + getPortString() + "/" + dbName; + + case elastic: + return "https://" + host + getPortString() + "/" + dbName; + + case file: // h2: + return "file:" + dbName; + } + throw new RuntimeException("Unsupported DBType: " + dbType); + } + + private String getPortString() { + return port > 0 ? (":" + Integer.toString(port)) : ""; + } + + /** + * Return BSim server info in URL + * @return BSim server info in URL + * @throws MalformedURLException if unable to form supported URL + */ + public URL toURL() throws MalformedURLException { + return new URL(toURLString()); + } + + /** + * @return BSim database type + */ + public DBType getDBType() { + return dbType; + } + + /** + * Get the server hostname or IP address as originally specified. + * @return hostname or IP address as originally specified + */ + public String getServerName() { + return host; + } + + /** + * Get the port number. + * @return port number + */ + public int getPort() { + return port; + } + + /** + * Get the DB Name + * @return DB name + */ + public String getDBName() { + return dbName; + } + + /** + * Get the DB Name. In the case of {@link DBType#file} the directory path will + * be excluded from returned name. + * @return shortened DB Name + */ + public String getShortDBName() { + if (shortDbName != null) { + return shortDbName; + } + shortDbName = dbName; + if (dbType == DBType.file) { + int ix = dbName.lastIndexOf('/'); + if (ix >= 0) { + shortDbName = dbName.substring(ix + 1); + } + } + return shortDbName; + } + + @Override + public int hashCode() { + // use dbType.ordinal; enum hashcodes vary from run to run + return Objects.hash(dbName, dbType.ordinal(), host, port); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) + return true; + if (obj == null) + return false; + if (obj instanceof BSimServerInfo other) { + return Objects.equals(dbName, other.dbName) && dbType == other.dbType && + Objects.equals(host, other.host) && port == other.port; + } + return false; + } + + @Override + public String toString() { + switch (dbType) { + case file: + return getShortDBName() + " (" + dbName + ");"; + default: + return dbName + " (" + dbType + ": " + host + ")"; + } + } + + /** + * Get a BSim {@link FunctionDatabase} instance which corresponds to this DB server info. + * The {@link Closeable} instance should be closed when no longer in-use to ensure that + * any associated database connection and resources are properly closed. + * @param async true if database commits should be asynchronous (may not be applicable) + * @return BSim function database instance + */ + public FunctionDatabase getFunctionDatabase(boolean async) { + return BSimClientFactory.buildClient(this, async); + } + + @Override + public int compareTo(BSimServerInfo o) { + return toString().compareTo(o.toString()); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BsimPluginPackage.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BsimPluginPackage.java new file mode 100644 index 0000000000..5f201de513 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/BsimPluginPackage.java @@ -0,0 +1,37 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import ghidra.framework.plugintool.util.PluginPackage; +import ghidra.framework.plugintool.util.PluginStatus; +import resources.ResourceManager; + +public class BsimPluginPackage extends PluginPackage { + + public static final String NAME = "BSim"; + + public BsimPluginPackage() { + super(NAME, ResourceManager.loadImage("images/preferences-web-browser-shortcuts-32.png"), + "An API and set of plugins for creating, managing and accessing function by similarity", + FEATURE_PRIORITY); + } + + @Override + public PluginStatus getActivationLevel() { + // bsim allows 'released' and 'stable' plugins for now + return PluginStatus.STABLE; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ChildMatchRecord.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ChildMatchRecord.java new file mode 100755 index 0000000000..e29efbefd6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ChildMatchRecord.java @@ -0,0 +1,63 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import generic.lsh.vector.LSHVector; +import ghidra.features.bsim.gui.search.results.BSimMatchResult; + +public class ChildMatchRecord implements Comparable{ + + BSimMatchResult similarFunction; + LSHVector vecWithChildren; + double significanceWithChildren; + double similarityWithChildren; + + + public ChildMatchRecord(BSimMatchResult match, LSHVector vec){ + this.similarFunction = match; + this.vecWithChildren = vec; + } + + public BSimMatchResult getSimilarFunction(){ + return this.similarFunction; + } + + public LSHVector getVecWithChildren(){ + return this.vecWithChildren; + } + + public void setSignificanceWithChildren(double newSignif){ + this.significanceWithChildren = newSignif; + } + + public double getSignificanceWithChildren(){ + return this.significanceWithChildren; + } + + public void setSimilarityWithChildren(double newSim){ + this.similarityWithChildren = newSim; + } + + public double getSimilarityWithChildren(){ + return this.similarityWithChildren; + } + + @Override + public int compareTo(ChildMatchRecord arg0) { + return Double.compare(arg0.getSignificanceWithChildren(), this.significanceWithChildren); + } + + } diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/CompareSignatures.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/CompareSignatures.java new file mode 100755 index 0000000000..13c0183fff --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/CompareSignatures.java @@ -0,0 +1,160 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +// Read in a set of precomputed signature files, generate a similarity score for each pair of functions +// listed in the files, if the similarity exceeds a threshold, print out a line to a file named "output" +import java.io.*; +import java.text.NumberFormat; +import java.util.Iterator; + +import org.xml.sax.ErrorHandler; +import org.xml.sax.SAXException; + +import generic.lsh.vector.*; +import ghidra.features.bsim.query.description.*; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +public class CompareSignatures { + + private DescriptionManager manager; + private LSHVectorFactory vectorFactory; + + public CompareSignatures(LSHVectorFactory vFactory) { + manager = new DescriptionManager(); + vectorFactory = vFactory; + } + + private boolean isFileSignatures(File file) { + if (file == null) return false; + try { + BufferedReader in = new BufferedReader(new FileReader(file)); + String line = in.readLine(); + in.close(); + if (line == null) return false; + if (line.contains("")) return true; + } + catch (FileNotFoundException e) { + return false; + } + catch (IOException e) { + return false; + } + return false; + } + + private void readFiles(File directory) { + ErrorHandler handler = SpecXmlUtils.getXmlHandler(); + File[] filelist = directory.listFiles(); + for (File element : filelist) { + if (isFileSignatures(element)) { + try { + XmlPullParser parser = new NonThreadedXmlPullParserImpl(element,handler,false); + manager.restoreXml(parser,vectorFactory); + } + catch (SAXException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + catch (IOException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } catch (LSHException e) { + // TODO Auto-generated catch block + e.printStackTrace(); + } + } + } + + } + + private void printResultRow(PrintStream out,double sim,double signif,FunctionDescription func1,FunctionDescription func2) { + NumberFormat nf = NumberFormat.getInstance(); + nf.setMaximumFractionDigits(2); + nf.setMinimumFractionDigits(2); + out.print(nf.format(sim)); + out.print(" "); + out.print(nf.format(signif)); + out.print(" "); + ExecutableRecord exerec = func1.getExecutableRecord(); + out.print(exerec.getNameExec()); + out.print(':'); + out.print(func1.getFunctionName()); + out.print(" "); + exerec = func2.getExecutableRecord(); + out.print(exerec.getNameExec()); + out.print(':'); + out.println(func2.getFunctionName()); + } + + private void compareSignatures(PrintStream out,double simthresh,double signifthresh) { + Iterator iter1,iter2; + VectorCompare veccompare = new VectorCompare(); + iter1 = manager.listAllFunctions(); + while(iter1.hasNext()) { + FunctionDescription func1 = iter1.next(); + LSHVector vec1 = func1.getSignatureRecord().getLSHVector(); + iter2 = manager.listFunctionsAfter(func1); + while(iter2.hasNext()) { + FunctionDescription func2 = iter2.next(); + LSHVector vec2 = func2.getSignatureRecord().getLSHVector(); + double res = vec1.compare(vec2,veccompare); + if (res > simthresh) { + double signif = vectorFactory.calculateSignificance(veccompare); + if (signif > signifthresh) + printResultRow(out,res,signif,func1,func2); + } + } + } + } + + + public void run(String[] args) { + if ((args==null)||(args.length<1)) { + System.out.println("Require signature directory path"); + } + File directory = new File(args[0]); + + double simthresh = 0.7; // Threshold for similarity + double signifthresh = 4.0; // Threshold for significance (close to size of function) + + readFiles(directory); + try { + PrintStream out; + if (args.length > 1) { + File outfile = new File(args[1]); + out = new PrintStream(new FileOutputStream(outfile)); + compareSignatures(out,simthresh,signifthresh); + out.close(); + } + else { + compareSignatures(System.out,simthresh,signifthresh); + } + } + catch (IOException e) { + e.printStackTrace(); + } + } + + public static void main(String[] args) { + LSHVectorFactory vFactory = new WeightedLSHCosineVectorFactory(); + // TODO: We need to load a weight file here + CompareSignatures comp = new CompareSignatures(vFactory); + comp.run(args); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/DecompileFunctionTask.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/DecompileFunctionTask.java new file mode 100755 index 0000000000..04063af7d6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/DecompileFunctionTask.java @@ -0,0 +1,37 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import ghidra.app.decompiler.DecompileException; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.util.task.TaskMonitor; + +/** + * Interface for a task that is initialized with a program, and a number of workers. + * Then the task is replicated for that number of workers, and each replicated task + * has its -decompile- method called some number of times with different functions + * and produces some output object + */ +public interface DecompileFunctionTask { + public void initializeGlobal(Program program); + + public DecompileFunctionTask clone(int worker) throws DecompileException; + + public void decompile(Function func, TaskMonitor monitor); + + public void shutdown(); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/FunctionDatabase.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/FunctionDatabase.java new file mode 100755 index 0000000000..c9e2967107 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/FunctionDatabase.java @@ -0,0 +1,424 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.io.*; +import java.util.*; + +import javax.xml.parsers.*; + +import org.w3c.dom.Document; +import org.w3c.dom.Element; +import org.xml.sax.SAXException; + +import generic.jar.ResourceFile; +import generic.lsh.vector.LSHVectorFactory; +import generic.lsh.vector.WeightedLSHCosineVectorFactory; +import ghidra.features.bsim.query.client.Configuration; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.facade.SFOverviewInfo; +import ghidra.features.bsim.query.facade.SFQueryInfo; +import ghidra.features.bsim.query.protocol.*; +import ghidra.framework.Application; +import ghidra.util.Msg; + +public interface FunctionDatabase extends AutoCloseable { + + public enum Status { + Unconnected("Unconnected"), Busy("Busy"), Error("Error"), Ready("Ready"); + + private final String label; + + private Status(String label) { + this.label = label; + } + + @Override + public String toString() { + return label; + } + } + + public enum ConnectionType { + // TODO: Should we add Never_Connected ? + SSL_No_Authentication(0), SSL_Password_Authentication(1), Unencrypted_No_Authentication(2); + + private ConnectionType(int label) { + // label is currently unused + } + } + + public enum ErrorCategory { + Unused(0), + Nonfatal(1), + Fatal(2), + Initialization(3), + Format(4), + Nodatabase(5), + Connection(6), + Authentication(7), + AuthenticationCancelled(8); + + private final int label; + + private ErrorCategory(int label) { + this.label = label; + } + + public int getInteger() { + return label; + } + } + + public static class Error { // Error structure returned by getLastError + public ErrorCategory category; + public String message; + + public Error(ErrorCategory cat, String msg) { + category = cat; + message = msg; + } + + @Override + public String toString() { + return message; + } + } + + public static class DatabaseNonFatalException extends Exception { + private static final long serialVersionUID = 1L; + + public DatabaseNonFatalException(String message) { + super(message); + } + } + + /** + * Determine if the connected database supports a user password change. + * @return true if a password change is permitted, else false. + */ + public default boolean isPasswordChangeAllowed() { + return getStatus() == Status.Ready && + getConnectionType() == ConnectionType.SSL_Password_Authentication; + } + + /** + * Issue password change request to the server. + * The method {@link #isPasswordChangeAllowed()} must be invoked first to ensure that + * the user password may be changed. + * @param username to change + * @param newPassword is password data + * @return null if change was successful, or the error message + */ + public default String changePassword(String username, char[] newPassword) { + if (getStatus() != Status.Ready) { + return "Connection not established"; + } + if (!isPasswordChangeAllowed()) { + return "Password change not supported"; + } + PasswordChange passwordChange = new PasswordChange(); + try { + passwordChange.username = username; + passwordChange.newPassword = newPassword; + ResponsePassword response = passwordChange.execute(this); + if (!response.changeSuccessful) { + return response.errorMessage; + } + } + finally { + passwordChange.clearPassword(); + } + return null; + } + + /** + * @return the status of the current connection with this database + */ + public Status getStatus(); + + /** + * @return the type of connection + */ + public ConnectionType getConnectionType(); + + /** + * @return username (being used to establish connection) + */ + public String getUserName(); + + /** + * Set a specific user name for connection. Must be called before connection is initialized. + * If this method is not called, connection will use user name of process + * + * @param userName the user name + */ + public void setUserName(String userName); + + /** + * @return factory the database is using to create LSHVector objects + */ + public LSHVectorFactory getLSHVectorFactory(); + + /** + * @return an information object giving general characteristics and descriptions of this database + */ + public DatabaseInformation getInfo(); + + /** + * Return -1 if info layout version is earlier than current client expectation + * Return 1 if info layout version is later than current client expectation + * Return 0 if info version and client version are the same + * @return comparison of actual database layout with layout expected by client + */ + public int compareLayout(); + + /** + * Return the {@link BSimServerInfo server info object} for this database + * @return the server info object + */ + public BSimServerInfo getServerInfo(); + + /** + * Get the + * @return + */ + @Deprecated + public String getURLString(); + + /** + * Initialize (a connection with) the database. If initialization is not successful, this routine will + * return false and an error description can be obtained using getLastError + * @return true if the database ready for querying + */ + public boolean initialize(); + + /** + * Close down (the connection with) the database + */ + @Override + public void close(); + + /** + * If the last query failed to produce a response, use this method to recover the error message + * @return a String describing the error + */ + public Error getLastError(); + + /** + * Send a query to the database. The response is returned as a QueryResponseRecord. + * If this is null, an error has occurred and an error message can be obtained from getLastError + * @param query an object describing the query + * @return the response object or null if there is an error + */ + public QueryResponseRecord query(BSimQuery query); + + public static void checkSettingsForQuery(DescriptionManager manage, DatabaseInformation info) + throws LSHException { + final int res = info.checkSignatureSettings(manage.getMajorVersion(), + manage.getMinorVersion(), manage.getSettings()); + if (res <= 1) { + return; + } + if (res == 4) { + return; + } + if (res == 3) { + throw new LSHException("Query signature data has no setting information"); + } + throw new LSHException("Query signature data does not match database"); + } + + public static boolean checkSettingsForInsert(DescriptionManager manage, + DatabaseInformation info) throws LSHException, DatabaseNonFatalException { + if (manage.numFunctions() == 0) { + throw new DatabaseNonFatalException("ls ~/junk" + ""); + } + int res = info.checkSignatureSettings(manage.getMajorVersion(), manage.getMinorVersion(), + manage.getSettings()); + if (res == 0) { + return false; + } + if (res == 1) { + throw new LSHException( + "Trying to insert signature data with slight differences in settings"); + } + if (res == 4) { + return true; // This apparently is the first insert + } + if (res == 3) { + throw new LSHException("Trying to insert signature data with no setting information"); + } + throw new LSHException( + "Trying to insert signature data with settings that don't match database"); + } + + public static String constructFatalError(int flags, ExecutableRecord newrec, + ExecutableRecord orig) { + String res = null; + if ((flags & ExecutableRecord.METADATA_ARCH) != 0) { + res = newrec.getNameExec() + " already ingested with different architecture field: " + + orig.getArchitecture(); + } + else if ((flags & ExecutableRecord.METADATA_COMP) != 0) { + res = newrec.getNameExec() + " already ingested with different compiler field: " + + orig.getNameCompiler(); + } + else if ((flags & ExecutableRecord.METADATA_LIBR) != 0) { + res = newrec.getNameExec() + " already ingested -- library field differs!!"; + } + else if ((flags & ExecutableRecord.METADATA_REPO) != 0) { + res = newrec.getNameExec() + " already ingested from a different repository: " + + orig.getRepository(); + } + return res; + } + + public static String constructNonfatalError(int flags, ExecutableRecord newrec, + ExecutableRecord orig) { + String res; + if ((flags & ExecutableRecord.METADATA_NAME) != 0) { + res = newrec.getNameExec() + " already ingested with a different name: " + + orig.getNameExec(); + } + else if ((flags & ExecutableRecord.METADATA_PATH) != 0) { + res = newrec.getNameExec() + " already ingested under a different path: " + + orig.getPath(); + } + else if ((flags & ExecutableRecord.METADATA_DATE) != 0) { + res = newrec.getNameExec() + " already ingested with a different date: " + + orig.getDate().toString(); + } + else { + res = newrec.getNameExec() + " already ingested with UNKNOWN difference in metadata"; + } + return res; + } + + public static Configuration loadConfigurationTemplate(String configname) throws LSHException { + ResourceFile moduleDataSubDirectory; + final Configuration config = new Configuration(); + try { + moduleDataSubDirectory = Application.getModuleDataSubDirectory(""); + config.loadTemplate(moduleDataSubDirectory, configname); + } + catch (final FileNotFoundException e) { + throw new LSHException("Missing configuration data: " + e.getMessage()); + } + catch (final IOException e) { + throw new LSHException("Could open module data directory"); + } + catch (final SAXException e) { + throw new LSHException("Unable to parse configuration template"); + } + return config; + } + + /** + * Central location for building vector factory used by FunctionDatabase + * @return the LSHVectorFactory object + */ + public static WeightedLSHCosineVectorFactory generateLSHVectorFactory() { + return new WeightedLSHCosineVectorFactory(); + } + + /** + * Returns a list of all configuration template files. + * + * @return list of template files + */ + public static List getConfigurationTemplates() { + List templateFiles = new ArrayList<>(); + + ResourceFile moduleDataSubDirectory; + try { + moduleDataSubDirectory = Application.getModuleDataSubDirectory(""); + File templateDir = new File(moduleDataSubDirectory.getAbsolutePath()); + if (!templateDir.exists()) { + return Collections.emptyList(); + } + + FilenameFilter nameFilter = (dir, name) -> { + if (!name.endsWith(".xml")) { + return false; + } + + return true; + }; + + File[] files = templateDir.listFiles(nameFilter); + if (files != null) { + for (File file : files) { + if (isConfigTemplate(file)) { + templateFiles.add(file); + } + } + } + } + catch (IOException e) { + Msg.error(null, "Error retrieving configuration templates", e); + } + + return templateFiles; + } + + /** + * Determines if a given xml file is a config template. This is done by opening the file + * and checking for the presence of a root tag. + * + * @param file the file to inspect + * @return true if the file is config template + */ + static boolean isConfigTemplate(File file) { + + try { + DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); + DocumentBuilder builder = factory.newDocumentBuilder(); + Document doc = builder.parse(file); + + Element rootElem = doc.getDocumentElement(); + if (rootElem.getTagName().equals("dbconfig")) { + return true; + } + } + catch (ParserConfigurationException | SAXException | IOException e) { + Msg.error(null, "Error inspecting xml file", e); + } + + return false; + } + + /** + * Get the maximum number of functions to be queried per staged query when searching + * for similar functions. + * @return maximum number of functions to be queried per staged query, or 0 for default + * which is generally ten (10) per stage. See {@link SFQueryInfo#DEFAULT_QUERIES_PER_STAGE}. + */ + public default int getQueriedFunctionsPerStage() { + return 0; + } + + /** + * Get the maximum number of functions to be queried per staged query when performing + * an overview query. + * @return maximum number of functions to be queried per staged query, or 0 for default + * which is generally ten (10) per stage. See {@link SFOverviewInfo#DEFAULT_QUERIES_PER_STAGE}. + */ + public default int getOverviewFunctionsPerStage() { + return 0; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/GenSignatures.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/GenSignatures.java new file mode 100755 index 0000000000..36cadc1826 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/GenSignatures.java @@ -0,0 +1,741 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.io.IOException; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.text.ParseException; +import java.text.SimpleDateFormat; +import java.util.*; +import java.util.concurrent.ConcurrentLinkedDeque; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.function.BiPredicate; + +import generic.jar.ResourceFile; +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.decompiler.*; +import ghidra.app.decompiler.signature.SignatureResult; +import ghidra.features.bsim.gui.filters.FunctionTagBSimFilterType; +import ghidra.features.bsim.query.client.AbstractSQLFunctionDatabase; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.PreFilter; +import ghidra.framework.Application; +import ghidra.framework.model.DomainFile; +import ghidra.framework.options.Options; +import ghidra.program.model.address.Address; +import ghidra.program.model.lang.LanguageID; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.*; +import ghidra.util.Msg; +import ghidra.util.task.TaskMonitor; + +/** + * Generate decompiler signatures for a set of functions + * + */ +public class GenSignatures { + private DescriptionManager manager; + private LSHVectorFactory vectorFactory; + private Program program; // Current program being analyzed + private FunctionManager fmanage; + private DecompileOptions options; + private ExecutableRecord exerec; + private SignatureTask singletask; // Task for processing one function at a time + private HashMap attributes; // Attributes to associate with functions + private List categories; // Category types associated with executables + private String dateColumnName; + private boolean gencallgraph; // True if callgraph info should be generated along with signatures + + private AtomicBoolean isShutdown = new AtomicBoolean(false); + private ConcurrentLinkedDeque runningTasks = + new ConcurrentLinkedDeque<>(); + + /** + * Prepare for generation of signature information and (possibly) callgraph information + * @param callgraph is true if the user wants callgraph information to be generated at the same time as signatures + */ + public GenSignatures(boolean callgraph) { + vectorFactory = null; + options = null; + exerec = null; + singletask = null; + gencallgraph = callgraph; + + attributes = new HashMap(); + attributes.put("Function ID Analyzer", FunctionTagBSimFilterType.KNOWN_LIBRARY_MASK); + categories = null; + dateColumnName = null; + } + + public void addExecutableCategories(List names) { + if (names == null) { + return; + } + for (String name : names) { + if (!CategoryRecord.enforceTypeCharacters(name)) { + throw new IllegalArgumentException(); + } + if (categories == null) { + categories = new ArrayList(); + } + categories.add(name); + } + } + + public void addFunctionTags(List names) { + if (names == null) { + return; + } + int flag = 1; + flag <<= FunctionTagBSimFilterType.RESERVED_BITS; // The first bits are reserved + for (String name : names) { + if ((flag == 0) || (!CategoryRecord.enforceTypeCharacters(name))) { + throw new IllegalArgumentException(); + } + attributes.put(name, flag); + flag <<= 1; + } + } + + public void setVectorFactory(LSHVectorFactory vFactory) throws LSHException { + if (vFactory.getSettings() == 0) { + throw new LSHException("Cannot have signature setting of 0"); + } + if (vectorFactory == vFactory) { // No change to factory + return; + } + if (manager != null) { + manager.clearFunctions(); // Clear out cached signature as settings have changed + } + vectorFactory = vFactory; + } + + public void addDateColumnName(String name) { + if (name == null) { + return; + } + if (!CategoryRecord.enforceTypeCharacters(name)) { + throw new IllegalArgumentException(); + } + dateColumnName = name; + } + + public DescriptionManager getDescriptionManager() { + return manager; + } + + /** + * Clear out any accumulated signatures + */ + public void clear() { + if (manager != null) { + manager.clear(); + } + manager = null; + program = null; // Cannot reuse unless we call openProgram again + } + + /** + * Generate an MD5 hash based just on executable metadata + * @param nmover is name of executable + * @param compover is architecture metadata + * @param archover is architecture metadata + * @return the md5 result as an ascii string + */ + private String generateMetadataMD5(String nmover, String compover, String archover) { + // until we can get full hash + MessageDigest digester = null; + try { + digester = MessageDigest.getInstance("MD5"); + } + catch (NoSuchAlgorithmException e) { + e.printStackTrace(); + } + byte data[] = new byte[nmover.length() + compover.length() + archover.length()]; + int pos = 0; + for (int i = 0; i < nmover.length(); ++i) { + data[pos++] = (byte) nmover.charAt(i); + } + for (int i = 0; i < compover.length(); ++i) { + data[pos++] = (byte) compover.charAt(i); + } + for (int i = 0; i < archover.length(); ++i) { + data[pos++] = (byte) archover.charAt(i); + } + digester.update(data); + byte[] digest = digester.digest(); + StringBuffer buf = new StringBuffer(); + char[] hexdigits = + { '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', 'a', 'b', 'c', 'd', 'e', 'f' }; + for (int i = 0; i < 16; ++i) { + char val1 = hexdigits[(digest[i] >> 4) & 0xf]; + char val2 = hexdigits[digest[i] & 0xf]; + buf.append(val1).append(val2); + } + return buf.toString(); + } + + /** + * Prepare to collect signatures for a new program, essentially by starting up a new decompiler process + * and creating an ExecutableRecord + * @param prog is the program to prepare for + * @param nmover if not null, overrides the "name" of the executable + * @param archover if not null, overrides the "architecture" of the executable + * @param compover if not null, overrides the "compiler" used to build the executable + * @param repo the repository containing the executable + * @param path the path (within the repo) where the executable can be found + * @throws LSHException if a new executable record cannot be created + */ + public void openProgram(Program prog, String nmover, String archover, String compover, + String repo, String path) throws LSHException { + program = prog; + + if (nmover == null) { + nmover = program.getDomainFile().getName(); + } + if (archover == null) { + archover = program.getLanguageID().getIdAsString(); + } + if (compover == null) { + compover = program.getCompilerSpec().getCompilerSpecID().getIdAsString(); + } + + String md5string = prog.getExecutableMD5(); + if ((md5string == null) || (md5string.length() < 10)) { + md5string = generateMetadataMD5(nmover, compover, archover); + } + Date progDate = fillinDate(); + manager = new DescriptionManager(); + exerec = manager.newExecutableRecord(md5string, nmover, compover, archover, progDate, repo, + path, null); + singletask = null; // Throw out any old decompiler process + fmanage = program.getFunctionManager(); + fillinExecutableCategories(); + } + + private void fillinExecutableCategories() { + if (categories == null) { + return; + } + List catrecs = new ArrayList(); + Options progoptions = program.getOptions(Program.PROGRAM_INFO); + for (String cat : categories) { // Search for each of the categories we want to record + String curproperty = cat; + int count = 0; + while (progoptions.contains(curproperty)) { + Object optionobject = progoptions.getObject(curproperty, null); + if (optionobject instanceof String) { + CategoryRecord rec = new CategoryRecord(cat, (String) optionobject); + catrecs.add(rec); + } + else { + break; + } + count += 1; + curproperty = cat + '_' + Integer.toString(count); + } + } + if (!catrecs.isEmpty()) { + manager.setExeCategories(exerec, catrecs); + } + } + + private Date fillinDate() { + if ((dateColumnName == null) || dateColumnName.equals(Program.DATE_CREATED) || + dateColumnName.equals("Ingest Date")) { + return program.getCreationDate(); + } + Options progoptions = program.getOptions(Program.PROGRAM_INFO); + if (!progoptions.contains(dateColumnName)) { + return ExecutableRecord.EMPTY_DATE; + } + Object optionobject = progoptions.getObject(dateColumnName, null); + if (optionobject instanceof Date) { + return (Date) optionobject; + } + Date res = ExecutableRecord.EMPTY_DATE; + if (optionobject instanceof String) { + String str = (String) optionobject; + if (str.length() == 19) { + SimpleDateFormat dateFormat = + new SimpleDateFormat(AbstractSQLFunctionDatabase.JAVA_TIME_FORMAT); + try { + res = dateFormat.parse(str); + } + catch (ParseException e) { + res = ExecutableRecord.EMPTY_DATE; + } + } + } + return res; + } + + private CallRecord fillinProperties(Address addr) { + CallRecord callRecord = new CallRecord(); + Function func = fmanage.getReferencedFunction(addr); + Symbol rootSymbol = null; + Address rootAddr = null; + boolean hasbody = false; + if (func == null) { // Found no function at all + SymbolTable symtab = program.getSymbolTable(); + rootSymbol = symtab.getPrimarySymbol(addr); // Look for any primary symbol + } + else { + if (func.isThunk()) { // Function looks like a thunk + func = func.getThunkedFunction(true); + } + rootAddr = func.getEntryPoint(); + rootSymbol = func.getSymbol(); + if (!func.isExternal()) { + hasbody = hasBody(rootAddr); + } + } + if (hasbody) { // Internal call + callRecord.exerec = exerec; // Within the same executable + callRecord.address = rootAddr.getOffset(); + callRecord.funcname = rootSymbol.getName(true); + } + else { // Treat as external call + callRecord.address = -1; // Address not available, indicate an external call + String libraryName; + if (rootSymbol == null) { + libraryName = "unknown"; + callRecord.funcname = "func_" + Long.toHexString(addr.getOffset()); // Make up a name + } + else { + libraryName = extractExternalName(rootSymbol, callRecord); + } + try { + callRecord.exerec = + manager.newExecutableLibrary(libraryName, exerec.getArchitecture(), null); + } + catch (LSHException e) { + callRecord.exerec = exerec; // If we couldn't create a library executable, use original executable + } + } + return callRecord; + } + + private String extractExternalName(Symbol sym, CallRecord callRecord) { + String fullName = sym.getName(true); + int ind = fullName.indexOf("::"); + String libraryName; + if (ind >= 0) { + libraryName = fullName.substring(0, ind); // First namespace is name of library + String tmpnm = fullName.substring(ind + 2); // Cut off first namespace + if (tmpnm.isEmpty()) { + libraryName = "unknown"; + callRecord.funcname = fullName; + } + else if (libraryName.isEmpty()) { + libraryName = "unknown"; + callRecord.funcname = tmpnm; + } + else { + callRecord.funcname = tmpnm; + } + if (libraryName.equals(Library.UNKNOWN)) { + libraryName = "unknown"; + } + } + else { + libraryName = "unknown"; + callRecord.funcname = fullName; + } + return libraryName; + } + + private int recoverAttributes(Function func) { + int flags = 0; + BookmarkManager bookmarkManager = program.getBookmarkManager(); + Bookmark[] bookmarks = bookmarkManager.getBookmarks(func.getEntryPoint()); + for (Bookmark bookmark : bookmarks) { + Integer val = attributes.get(bookmark.getCategory()); + if (val != null) { + flags |= val.intValue(); + } + } + Set tags = func.getTags(); + for (FunctionTag tag : tags) { + Integer val = attributes.get(tag.getName()); + if (val != null) { + flags |= val.intValue(); + } + } + return flags; + } + + private List collectCallsFromAddress(List

calladdr) { + List calls = new ArrayList(); + for (int i = 0; i < calladdr.size(); ++i) { + Address addr = calladdr.get(i); + CallRecord callRecord = fillinProperties(addr); + calls.add(callRecord); + } + return calls; + } + + /** + * Return true if the address corresponds to a normal function body + * @param addr is the entry point of the function + * @return true if it has a body + */ + private boolean hasBody(Address addr) { + Listing listing = program.getListing(); + CodeUnit cu = listing.getCodeUnitAt(addr); + if (!(cu instanceof Instruction)) { // If the entry point is data -> no body + return false; + } + + Instruction inst = (Instruction) cu; + FlowType flowType = inst.getFlowType(); + if (flowType == RefType.COMPUTED_JUMP) { + return false; + } + + return true; + } + + private synchronized void writeToManager(Function func, int[] hash, List callrecs, + int flags) { + FunctionDescription fdesc = manager.newFunctionDescription(func.getName(true), + func.getEntryPoint().getOffset(), exerec); + manager.setFunctionDescriptionFlags(fdesc, flags); + if (hash != null) { + LSHVector vec = vectorFactory.buildVector(hash); + SignatureRecord sigrec = manager.newSignature(vec, 0); + manager.attachSignature(fdesc, sigrec); + } + for (CallRecord callRecord : callrecs) { + FunctionDescription destfunc = manager.newFunctionDescription(callRecord.funcname, + callRecord.address, callRecord.exerec); + manager.makeCallgraphLink(fdesc, destfunc, 0); + } + } + + public int transferCachedFunctions(DescriptionManager otherman, Iterator functions, + PreFilter preFilter) throws LSHException { + otherman.transferSettings(manager); + int count = 0; + BiPredicate filterPredicate = + preFilter.getAndReducedPredicate(); + while (functions.hasNext()) { + FunctionDescription desc; + Function func = functions.next(); + String name = func.getName(true); + long address = func.getEntryPoint().getOffset(); + try { + desc = manager.findFunction(name, address, exerec); + } + catch (LSHException e) { // This exception is thrown if the manager does not contain a function of this name + continue; // Basically we skip the function in this case + } + if (filterPredicate.test(program, desc)) { + otherman.transferFunction(desc, true); + count += 1; + } + } + return count; + } + + /** + * Generate signatures for a (potentially large) set of functions by spawning multiple + * threads to parallelize the work + * + * @param functions the set of functions to signature + * @param countestimate estimated number of functions (to initialize the monitor) + * @param monitor controls interruptions and progress reports + * @throws DecompileException if the functions cannot be decompiled + */ + public void scanFunctions(Iterator functions, int countestimate, TaskMonitor monitor) + throws DecompileException { + + if (!functions.hasNext()) { + return; + } + + if (isShutdown.get()) { + return; + } + + ParallelDecompileTask taskrun = + new ParallelDecompileTask(program, monitor, new SignatureTask()); + runningTasks.add(taskrun); + taskrun.decompile(functions, countestimate); + runningTasks.remove(taskrun); + } + + /** + * Calculate signatures for a single function + * @param func is the function to scan + * @throws DecompileException if the decompiler task fails + */ + public void scanFunction(Function func) throws DecompileException { + if (isShutdown.get()) { + return; + } + + if (singletask == null) { + singletask = new SignatureTask(); + singletask.initializeGlobal(program); + singletask = (SignatureTask) singletask.clone(0); // Start decompiler process + } + + singletask.decompile(func, null); + } + + /** + * Generate just the update metadata for functions in the currently open program + * if -iter- is null, generate metadata for all functions + * @param iter iterates over the set of Functions to generate metadata for + * @param monitor the task monitor + */ + public void scanFunctionsMetadata(Iterator iter, TaskMonitor monitor) { + if (exerec == null) { + return; // No current executable + } + if (iter == null) { + iter = fmanage.getFunctions(true); + } + while (iter.hasNext()) { + Function func = iter.next(); + if ((monitor != null) && (monitor.isCancelled())) { + return; + } + if (func.isThunk()) { + continue; + } + if (!hasBody(func.getEntryPoint())) { + continue; + } + int flags = recoverAttributes(func); + FunctionDescription fdesc = manager.newFunctionDescription(func.getName(true), + func.getEntryPoint().getOffset(), exerec); + manager.setFunctionDescriptionFlags(fdesc, flags); + } + } + + public void dispose() { + + isShutdown.set(true); + + for (ParallelDecompileTask task : runningTasks) { + task.shutdown(); + } + runningTasks.clear(); + + if (singletask != null) { + singletask.shutdown(); + } + + clear(); + } + + /** + * Build an ExecutableRecord path from the domain file. + * WARNING: Make sure the program has been saved previously before calling this, otherwise you get + * an (inaccurate) result of "/" + * @param program the current program + * @return the path to this program within the repository as a string + */ + public static String getPathFromDomainFile(Program program) { + DomainFile domainFile = program.getDomainFile(); + String path = domainFile.getPathname(); + int ind = path.length() - domainFile.getName().length(); + if (ind >= 0) { + if (!path.substring(ind).equals(domainFile.getName())) { + ind = -1; + } + } + if (ind <= 0) { + path = null; + } + else { + path = path.substring(0, ind); + } + return path; + } + + /** + * Return the weights file that should be used to compare functions between two programs + * @param id1 is the language of the first program + * @param id2 is the language of the second program (can be same as first program) + * @return the XML weights file, or null if there is no valid weights file + * @throws IOException if the module data directory cannot be found + */ + public static ResourceFile getWeightsFile(LanguageID id1, LanguageID id2) throws IOException { + String[] split1 = id1.getIdAsString().split(":"); + String[] split2 = id2.getIdAsString().split(":"); + // Check if we have are comparing the same processor size + if ((split1.length < 3) || (split2.length < 3)) { + return null; // If not, we need to do something different with the weights + } + + ResourceFile moduleDataSubDirectory = Application.getModuleDataSubDirectory(""); + if (split1[0].equals("Dalvik") || split1[0].equals("JVM")) { + if (!split2[0].equals(split1[0])) { + return null; + } + return new ResourceFile(moduleDataSubDirectory, "lshweights_cpool.xml"); + } + String basefile; + String size1 = split1[2]; // Pull out the size + String size2 = split2[2]; + if (!size1.equals(size2)) { // If the two things we compare are from different processor sizes + if (!size1.equals("64") && !size1.equals("32")) { + return null; // Differing sizes not 32 or 64 + } + if (!size2.equals("64") && !size2.equals("32")) { + return null; // We cannot do decent comparisons + } + basefile = "lshweights_nosize.xml"; // We use a special sizeless weights file + } + else { + if (size1.equals("32")) { + basefile = "lshweights_32.xml"; + } + else if (size1.equals("64")) { + basefile = "lshweights_64.xml"; + if (split1.length > 3) { + String version = split1[3]; + if (version.contains("-32")) { + basefile = "lshweights_64_32.xml"; + } + } + } + else { + // Same size, but not 64 or 32 + basefile = "lshweights_nosize.xml"; + } + } + + return new ResourceFile(moduleDataSubDirectory, basefile); + } + + /** + * Info for resolving a call to a unique function in the database. + * For normal functions you need the triple (executable, function name, address) + * For calls to library (external) functions, only the library executable + * and the function name are needed, and the address is filled in with -1 + */ + private static class CallRecord { + public ExecutableRecord exerec; + public String funcname; + public long address; + } + + public class SignatureTask implements DecompileFunctionTask { + + private DecompInterface decompiler; + + public SignatureTask() { + decompiler = null; + } + + private SignatureTask(DecompInterface decompiler) { + this.decompiler = decompiler; + } + + @Override + public DecompileFunctionTask clone(int worker) throws DecompileException { + DecompInterface newdecompiler = new DecompInterface(); + newdecompiler.setOptions(options); + newdecompiler.toggleSyntaxTree(false); + newdecompiler.setSignatureSettings(vectorFactory.getSettings()); + if (!newdecompiler.openProgram(program)) { + String errorMessage = newdecompiler.getLastMessage(); + throw new DecompileException("Decompiler", + "Unable to initialize the DecompilerInterface: " + errorMessage); + } + if (worker == 0) { // Query the first work for settings info + short major = newdecompiler.getMajorVersion(); + short minor = newdecompiler.getMinorVersion(); + int settings = newdecompiler.getSignatureSettings(); + manager.setVersion(major, minor); + manager.setSettings(settings); + } + return new SignatureTask(newdecompiler); + } + + @Override + public void decompile(Function func, TaskMonitor monitor) { + if ((monitor != null) && (monitor.isCancelled())) { + return; + } + if (func.isThunk()) { + return; + } + Address entryPoint = func.getEntryPoint(); + if (!hasBody(entryPoint)) { + return; + } + FunctionDescription fdesc = + manager.containsDescription(func.getName(true), entryPoint.getOffset(), exerec); + if (fdesc != null && fdesc.getSignatureRecord() != null) { // Is signature for this function already present + return; + } + SignatureResult sigres = decompiler.generateSignatures(func, gencallgraph, + options.getDefaultTimeout(), monitor); + + if ((monitor != null) && (monitor.isCancelled())) { + return; + } + + if (sigres == null) { + String errmsg = decompiler.getLastMessage(); + if ((errmsg != null) && !errmsg.isEmpty()) { + // throw new DecompileException("signature",errmsg); + Msg.error(this, "Error generating signature for \"" + func.getName() + + "\". Error: " + errmsg); + } + return; + } + else if (sigres.features.length == 0) { + Msg.error(this, "No features in signature for \"" + func.getName() + '\"'); + return; + } + int flags = recoverAttributes(func); + if (sigres.hasunimplemented) { + flags |= FunctionTagBSimFilterType.HAS_UNIMPLEMENTED_MASK; + } + if (sigres.hasbaddata) { + flags |= FunctionTagBSimFilterType.HAS_BADDATA_MASK; + } + List callrecs; + if (gencallgraph) { + callrecs = collectCallsFromAddress(sigres.calllist); + } + else { + callrecs = new ArrayList(); + } + writeToManager(func, sigres.features, callrecs, flags); + } + + @Override + public void initializeGlobal(Program prog) { + program = prog; + options = new DecompileOptions(); + options.grabFromProgram(program); // Same options are global across all tasks + } + + @Override + public void shutdown() { + decompiler.dispose(); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/LSHException.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/LSHException.java new file mode 100755 index 0000000000..f9c3323568 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/LSHException.java @@ -0,0 +1,30 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +public class LSHException extends Exception { + private static final long serialVersionUID = 1L; + + public LSHException(String msg) { + super(msg); + } + + @Override + public String toString() { + return "LSHException: " + getMessage(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/MinimalErrorLogger.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/MinimalErrorLogger.java new file mode 100755 index 0000000000..89d24ae8e4 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/MinimalErrorLogger.java @@ -0,0 +1,61 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import ghidra.util.DefaultErrorLogger; + +public class MinimalErrorLogger extends DefaultErrorLogger { + @Override + public void debug(Object originator, Object message) { + // Squash DEBUG messages + } + + @Override + public void debug(Object originator, Object message, Throwable throwable) { + // Squash DEBUG messages + } + + @Override + public void info(Object originator, Object message) { + // Squash INFO messages + } + + @Override + public void info(Object originator, Object message, Throwable throwable) { + // Squash INFO messages + } + + @Override + public void error(Object originator, Object message) { + System.err.println(message); + } + + @Override + public void error(Object originator, Object message, Throwable throwable) { + System.err.println(message); + // Don't print stack trace + } + + @Override + public void warn(Object originator, Object message) { + // Squash WARN messages + } + + @Override + public void warn(Object originator, Object message, Throwable throwable) { + // Squash WARN messages + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ParallelDecompileTask.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ParallelDecompileTask.java new file mode 100755 index 0000000000..93455422a1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ParallelDecompileTask.java @@ -0,0 +1,152 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.util.Iterator; + +import generic.cache.CachingPool; +import generic.cache.CountingBasicFactory; +import generic.concurrent.QCallback; +import ghidra.app.decompiler.DecompileException; +import ghidra.app.util.DecompilerConcurrentQ; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.util.Msg; +import ghidra.util.task.TaskMonitor; +import ghidra.util.task.TaskMonitorAdapter; + +/** + * Run decompilation across multiple functions in a single program, distributing the task across + * a specific number of threads + * + */ +public class ParallelDecompileTask { + private Program program; + private TaskMonitor taskMonitor = TaskMonitor.DUMMY; + private DecompileFunctionTask ftask_template; // Template of the worker task + + // save for shutting down abnormally + private DecompilerConcurrentQ queue; + + public ParallelDecompileTask(Program prog, TaskMonitor mon, DecompileFunctionTask ftask) { + program = prog; + if (mon != null) + taskMonitor = mon; + ftask_template = ftask; + + ftask_template.initializeGlobal(program); + } + + public void decompile(Iterator iter, int functionCount) throws DecompileException { + try { + doDecompile(iter, functionCount); + } + catch (InterruptedException e) { + Msg.error(this, "Problem with decompiler worker thread", e); + throw new DecompileException("interrupted", e.getMessage()); + } + catch (Exception t) { + Msg.error(this, "Problem with decompiler worker thread", t); + DecompileException decompileException = + new DecompileException("execution", t.getMessage()); + decompileException.initCause(t); + throw decompileException; + } + } + + private void doDecompile(Iterator iter, int functionCount) + throws InterruptedException, Exception { + taskMonitor.setMessage("Analyzing functions..."); + taskMonitor.initialize(functionCount); + + CachingPool decompilerPool = + new CachingPool(new DecompilerTaskFactory(ftask_template)); + QCallback callback = new ParallelDecompilerCallback(decompilerPool); + + queue = new DecompilerConcurrentQ(callback, taskMonitor); + + queue.addAll(iter); + try { + queue.waitUntilDone(); + } + finally { + queue.dispose(); + decompilerPool.dispose(); + } + } + + void shutdown() { + if (queue == null) { + return; + } + + // Wait, at least a bit, for the tasks to drop out of their work (we could be + // getting called from a tool shutdown event, which means we don't want to block + // indefinitely, or even too long. + + // for now use 5 seconds, which seems long when closing a tool, but the user did + // decide to exit without cancelling the task first, so it is reasonable to expect + // some delay + queue.dispose(5); + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + + private class ParallelDecompilerCallback implements QCallback { + + private CachingPool pool; + + ParallelDecompilerCallback(CachingPool decompilerPool) { + this.pool = decompilerPool; + } + + @Override + public Function process(Function func, TaskMonitor monitor) throws Exception { + monitor.setMessage("Decompiling " + func.getName()); + + DecompileFunctionTask task = pool.get(); + try { + task.decompile(func, monitor); + } + finally { + pool.release(task); + } + + return null; // we don't use results in the parallel implementation + } + } + + private class DecompilerTaskFactory extends CountingBasicFactory { + private DecompileFunctionTask taskFactory; + + DecompilerTaskFactory(DecompileFunctionTask taskFactory) { + this.taskFactory = taskFactory; + } + + @Override + public DecompileFunctionTask doCreate(int itemNumber) throws DecompileException { + int zeroBasedNumber = itemNumber - 1; + return taskFactory.clone(zeroBasedNumber); + } + + @Override + public void doDispose(DecompileFunctionTask task) { + task.shutdown(); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/SQLFunctionDatabase.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/SQLFunctionDatabase.java new file mode 100644 index 0000000000..15a5663c20 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/SQLFunctionDatabase.java @@ -0,0 +1,27 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +public interface SQLFunctionDatabase extends FunctionDatabase { + + /** + * Generate SQL bitwise-and syntax for use in database query WHERE clause + * @param v1 first value + * @param v2 second value + * @return SQL + */ + public String formatBitAndSQL(String v1, String v2); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ServerConfig.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ServerConfig.java new file mode 100755 index 0000000000..8d81512337 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ServerConfig.java @@ -0,0 +1,781 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query; + +import java.io.*; +import java.util.Map.Entry; +import java.util.TreeMap; +import java.util.TreeSet; + +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Class for modifying the PostgreSQL configuration files describing + * the main server settings (postgresql.conf) + * the connection settings (pg_hba.conf) + * the identification map (pg_ident.conf) + */ +public class ServerConfig { + private TreeMap keyValue = new TreeMap<>(); // Values we want set in the configuration file + private TreeSet connectSet = new TreeSet<>(); // Entries we want in the connection file + + /** + * Class that holds a single configuration option from the PostgreSQL configuration file + */ + private static class ConfigLine { + public String key; // Configuration key + public String value; // Value assigned to the key + public String comment; // Any associate comment on the same line as the key/value + public int status; + // 0 if the line does not contain a controlled key + // 1 if the line contains a controlled key that is uncommented + // 2 if the line contains a controlled key that is commented + + private int sz; + private int pos; + private boolean commentedkey; + + public void parseUptoKey(String line) { + key = null; + value = null; + comment = null; + status = 0; + sz = line.length(); + pos = 0; + commentedkey = false; + pos = skipWhiteSpace(line, pos); + if (pos >= sz) { + return; + } + if (line.charAt(pos) == '#') { + pos += 1; + commentedkey = true; + pos = skipWhiteSpace(line, pos); + if (pos >= sz) { + return; + } + } + int tokend = scanToken(line, pos); + if (tokend == pos) { + return; // No characters in token + } + key = line.substring(pos, tokend); + pos = tokend; + + } + + public void skipValueParseComment(String line) { + pos = skipWhiteSpace(line, pos); + if (pos >= sz) { + return; + } + if (line.charAt(pos) != '=') { + return; + } + pos += 1; + comment = ""; + + while (pos < sz) { + if (line.charAt(pos) == '#') { + comment = line.substring(pos); + break; + } + pos += 1; + } + status = commentedkey ? 2 : 1; + } + + public void parseValue(String line) { + pos = skipWhiteSpace(line, pos); + if (pos >= sz) { + return; + } + if (line.charAt(pos) != '=') { + return; + } + pos += 1; + comment = ""; + int valstart = pos; + while (pos < sz) { + if (line.charAt(pos) == '#') { + comment = line.substring(pos); + break; + } + pos += 1; + } + value = line.substring(valstart, pos).trim(); + status = commentedkey ? 2 : 1; + } + + private static int scanToken(String line, int pos) { + int sz = line.length(); + while (pos < sz) { + char c = line.charAt(pos); + if (!Character.isJavaIdentifierPart(c)) { + break; + } + pos += 1; + } + return pos; + } + + private static int skipWhiteSpace(String line, int pos) { + int sz = line.length(); + while (pos < sz) { + char c = line.charAt(pos); + if (!Character.isWhitespace(c)) { + break; + } + pos += 1; + } + return pos; + } + } + + /** + * Class that holds an entry from the PostgreSQL connection configuration file + */ + private static class ConnectLine implements Comparable { + public String type; // Type of connection: local, host, hostssl, etc. + public String database; // Name of database associated with entry or the reserved word 'all' + public String user; // Name of user associated with entry (or 'all') + public String address; // IPv4 or IPv6 address + public String method; // authentication method to use: trust, cert, ... + public String options; // Additional options + public boolean isMatched; // Set to true if we have seen this entry in the connection file + + @Override + public int compareTo(ConnectLine op2) { + int comp = database.compareTo(op2.database); + if (comp != 0) { + return comp; + } + comp = user.compareTo(op2.user); + if (comp != 0) { + return comp; + } + if (address == null) { + if (op2.address != null) { + return -1; + } + } + else { + if (op2.address == null) { + return 1; + } + comp = address.compareTo(op2.address); + if (comp != 0) { + return comp; + } + } + return 0; + } + + /** + * Determine if the connection is coming either from UNIX socket or "localhost" + * @return true if the connection is local in this sense + */ + public boolean isLocal() { + if (type.equals("local")) { // UNIX socket + return true; + } + if (address != null) { + if (address.equals("127.0.0.1/32")) { // IPv4 localhost + return true; + } + if (address.equals("::1/128")) { // IPv6 localhost + return true; + } + } + return false; + } + + /** + * Parse the fields out of a line of the connection file + * @param line the text to parse + * @throws IOException if the text is not formatted properly to parse + */ + public void parse(String line) throws IOException { + String[] split = line.split(" +"); // Split on whitespace + if (split.length < 4) { + throw new IOException("Parsing error"); + } + type = split[0]; + database = split[1]; + user = split[2]; + int nextPos = 3; + if (type.equals("local")) { // "local" type has no address + address = null; + } + else { + address = split[3]; + nextPos = 4; + } + if (nextPos >= split.length) { + throw new IOException("Parsing error"); + } + method = split[nextPos]; + nextPos += 1; + if (nextPos >= split.length) { + options = null; + return; + } + StringBuilder buffer = new StringBuilder(); + buffer.append(split[nextPos]); + nextPos += 1; + while (nextPos < split.length) { + buffer.append(' '); + buffer.append(split[nextPos]); + nextPos += 1; + } + options = buffer.toString(); + } + + /** + * Restore a connection entry from an XML tag + * @param el the XML element to restore + */ + public void restoreXml(XmlElement el) { + type = el.getAttribute("type"); + database = el.getAttribute("db"); + user = el.getAttribute("user"); + address = el.getAttribute("addr"); + method = el.getAttribute("method"); + options = el.getAttribute("options"); + } + + /** + * Emit the line, formatted as it should appear in the connection file + * @param writer the stream writer + * @throws IOException if appending to the stream fails + */ + public void emit(Writer writer) throws IOException { + writer.append(type); + for (int i = type.length(); i < 8; ++i) { + writer.append(' '); + } + writer.append(database); + for (int i = database.length(); i < 16; ++i) { + writer.append(' '); + } + writer.append(user); + for (int i = user.length(); i < 16; ++i) { + writer.append(' '); + } + int addrLen = 0; + if (address != null) { + addrLen = address.length(); + writer.append(address); + } + for (int i = addrLen; i < 24; ++i) { + writer.append(' '); + } + writer.append(method); + if (options != null) { + writer.append(' '); + writer.append(options); + } + } + } + + private static class IdentLine { + private String mapName; // Map the entry belongs to + private String systemName; // Name reported by the system + private boolean systemNameIsQuoted; + private String roleName; // Database role to map to + private boolean roleNameIsQuoted; + + private static int skipWhiteSpace(int pos, String line) { + for (; pos < line.length(); ++pos) { + char c = line.charAt(pos); + if (c != ' ' && c != '\t') { + break; + } + } + return pos; + } + + private static int parseField(int pos, String line) { + for (; pos < line.length(); ++pos) { + char c = line.charAt(pos); + if (c == ' ' || c == '\t') { + break; + } + } + return pos; + } + + private static int parseDoubleQuote(int pos, String line) { + pos += 1; // Skip the initial quote character + for (; pos < line.length(); ++pos) { + char c = line.charAt(pos); + if (c == '"') { + pos += 1; + break; + } + } + return pos; + } + + public static boolean needsDoubleQuotes(String name) { + for (int i = 0; i < name.length(); ++i) { + char c = name.charAt(i); + if (!Character.isLetterOrDigit(c)) { + return true; + } + } + return false; + } + + public IdentLine() { + } + + public IdentLine(String mName, String sysName, String rName) { + mapName = mName; + systemName = sysName; + systemNameIsQuoted = needsDoubleQuotes(sysName); + roleName = rName; + roleNameIsQuoted = needsDoubleQuotes(rName); + } + + public void setSystemName(String sysName) { + systemName = sysName; + systemNameIsQuoted = needsDoubleQuotes(sysName); + } + + public boolean matchRole(String mName, String rName) { + return mapName.equals(mName) && roleName.equals(rName); + } + + /** + * Parse a single line from the pg_ident.conf file and recover the + * map name, system name, and role + * @param line is the incoming of text + * @return true if the line is an ident entry, false if it is a comment + * @throws IOException if the text cannot be parsed + */ + public boolean parse(String line) throws IOException { + int pos = 0; + pos = skipWhiteSpace(pos, line); + if (pos >= line.length()) { + return false; // Blank line, treat as comment + } + if (line.charAt(pos) == '"') { + throw new IOException("Bad map field in pg_ident.conf entry"); + } + if (line.charAt(pos) == '#') { + return false; // Return false to indicate comment + } + int endpos = parseField(pos, line); + mapName = line.substring(pos, endpos); + + pos = skipWhiteSpace(endpos, line); + if (pos >= line.length()) { + throw new IOException("Missing system-name in pg_ident.conf entry"); + } + else if (line.charAt(pos) == '"') { + systemNameIsQuoted = true; + endpos = parseDoubleQuote(pos, line); + if (line.charAt(endpos - 1) != '"') { + throw new IOException("Entry missing ending quote in pg_ident.conf"); + } + systemName = line.substring(pos + 1, endpos - 1); // Strip quotes + } + else { + systemNameIsQuoted = false; + endpos = parseField(pos, line); + systemName = line.substring(pos, endpos); + } + + pos = skipWhiteSpace(endpos, line); + if (pos >= line.length()) { + throw new IOException("Missing role in pg_ident.conf entry"); + } + else if (line.charAt(pos) == '"') { + roleNameIsQuoted = true; + endpos = parseDoubleQuote(pos, line); + if (line.charAt(endpos - 1) != '"') { + throw new IOException("Entry missing ending quote in pg_ident.conf"); + } + roleName = line.substring(pos + 1, endpos - 1); // Strip quotes + } + else { + roleNameIsQuoted = false; + endpos = parseField(pos, line); + roleName = line.substring(pos, endpos); + } + return true; + } + + public void emit(Writer writer) throws IOException { + writer.write(mapName); + for (int i = mapName.length(); i < 15; ++i) { + writer.write(' '); + } + writer.write(' '); + if (systemNameIsQuoted) { + writer.write('"'); + } + writer.write(systemName); + if (systemNameIsQuoted) { + writer.write('"'); + } + for (int i = systemName.length() + (systemNameIsQuoted ? 2 : 0); i < 23; ++i) { + writer.write(' '); + } + writer.write(' '); + if (roleNameIsQuoted) { + writer.write('"'); + } + writer.write(roleName); + if (roleNameIsQuoted) { + writer.write('"'); + } + } + } + + /** + * Read a set of key/value pairs and connection entries to use for patching, from an XML file + * @param parser the XML parser + */ + public void restoreXml(XmlPullParser parser) { + parser.start("serverconfig"); + while (parser.peek().isStart()) { + XmlElement el = parser.start(); + if (el.getName().equals("config")) { + String key = el.getAttribute("key"); + String val = parser.end().getText(); + keyValue.put(key, val); + } + else if (el.getName().equals("connect")) { + ConnectLine connLine = new ConnectLine(); + connLine.isMatched = false; + connLine.restoreXml(el); + connectSet.add(connLine); + parser.end(el); + } + else { + parser.discardSubTree(el); + } + } + parser.end(); + } + + private static String stripConnectComment(String line) { + int pos = line.indexOf('#'); // Position of any comment character + if (pos == -1) { + pos = line.length(); + } + for (int i = 0; i < pos; ++i) { + char c = line.charAt(i); + if (c != ' ' && c != '\t') { // Is there meaning characters before the comment '#' character + return line.substring(i, pos); // Strip off comment + } + } + return null; // Indicate this line only contains spaces and/or comment + } + + /** + * Given a set of key/value pairs, established via restoreXml or manually entered via addKey, + * read in an existing configuration file, and write out an altered form, where: + * 1) Keys matching something in the keyValue map have their value altered to match the map + * 2) Keys that don't match anything in the map, are output unaltered + * 3) Comments, both entire line and those coming after key/value pairs, are preserved + * @param inFile the file to read + * @param outFile the new file to write + * @throws IOException if the files cannot be read from or written to + */ + public void patchConfig(File inFile, File outFile) throws IOException { + + TreeSet alreadyemitted = new TreeSet<>(); + String line; + ConfigLine parse = new ConfigLine(); + + BufferedReader reader = new BufferedReader(new FileReader(inFile)); + FileWriter writer = new FileWriter(outFile); + try { + for (;;) { + line = reader.readLine(); + if (line == null) { + break; // End of file reached + } + if (line.length() != 0) { + parse.parseUptoKey(line); + if (parse.key != null) { + parse.value = keyValue.get(parse.key); // Check if this is a key we control + } + if (parse.value != null) { + parse.skipValueParseComment(line); // Discard the original value, but preserve any comment + } + if (parse.status > 0) { // A controlled key + if (!alreadyemitted.contains(parse.key)) { // Have not emitted yet + line = parse.key + " = " + parse.value; + if (parse.comment.length() != 0) { + line = line + " " + parse.comment; + } + alreadyemitted.add(parse.key); + } + else { // Have already emitted before + if (parse.status == 1) { + line = '#' + line; + } + } + } + } + writer.write(line); + writer.write('\n'); + } + + for (Entry entry : keyValue.entrySet()) { + if (!alreadyemitted.contains(entry.getKey())) { + line = entry.getKey() + " = " + entry.getValue(); + writer.write(line); + writer.write('\n'); + } + } + } + finally { + reader.close(); + writer.close(); + } + } + + /** + * Read in a connection file and write out an altered version of the file where: + * 1) Any entry that matches something in connectSet, has its authentication method altered + * 2) Any entry that does not match into connectSet is commented out in the output + * 3) Entire line comments are preserved + * @param inFile the file to read + * @param outFile the new file to write + * @throws IOException if the files cannot be read from or written to + */ + public void patchConnect(File inFile, File outFile) throws IOException { + BufferedReader reader = new BufferedReader(new FileReader(inFile)); + FileWriter writer = new FileWriter(outFile); + try { + for (;;) { + String line = reader.readLine(); + if (line == null) { + break; // End of file reached + } + String stripLine = stripConnectComment(line); + if (stripLine == null) { // This line only contained a comment + writer.write(line); // Output the original line + } + else { + ConnectLine connLine = new ConnectLine(); + connLine.parse(stripLine); + ConnectLine matchLine = connectSet.ceiling(connLine); + if (matchLine == null || 0 != matchLine.compareTo(connLine)) { + writer.write('#'); // Comment out the line + connLine.emit(writer); + } + else { + matchLine.emit(writer); + matchLine.isMatched = true; + } + } + writer.write('\n'); + } + + // Append any entries we didn't match + for (ConnectLine connLine : connectSet) { + if (connLine.isMatched) { + continue; + } + connLine.emit(writer); + writer.write('\n'); + } + } + finally { + reader.close(); + writer.close(); + } + } + + /** + * Add/remove an identify entry to pg_ident.conf + * @param inFile is a copy of pg_ident.conf to modify + * @param outFile becomes the modified copy of pg_ident.conf + * @param mapName is the map being modified + * @param systemName is the system name (map from) + * @param roleName is the database role (map to) + * @param addUser is true if the map entry is to be added, false if the entry should be removed + * @throws IOException if the file cannot be read from or written to + */ + public static void patchIdent(File inFile, File outFile, String mapName, String systemName, + String roleName, boolean addUser) throws IOException { + BufferedReader reader = new BufferedReader(new FileReader(inFile)); + FileWriter writer = new FileWriter(outFile); + + try { + boolean entryIsMatched = !addUser; + for (;;) { + String line = reader.readLine(); + if (line == null) { + break; + } + IdentLine identLine = new IdentLine(); + if (identLine.parse(line)) { + if (identLine.matchRole(mapName, roleName)) { // Found old entry + if (!addUser) { // If we are supposed to drop the entry + continue; // Skip the emit method below + } + identLine.setSystemName(systemName); // Update to new role + entryIsMatched = true; + } + identLine.emit(writer); + writer.write('\n'); + } + else { // Read a comment + writer.write(line); // Keep line as is + writer.write('\n'); + } + } + if (!entryIsMatched) { + IdentLine identLine = new IdentLine(mapName, systemName, roleName); + identLine.emit(writer); + writer.write('\n'); + } + } + finally { + reader.close(); + writer.close(); + } + } + + /** + * Add a key/value pair directly into the configuration file + * @param key the key to add/update + * @param value the value to insert + */ + public void addKey(String key, String value) { + keyValue.put(key, value); + } + + /** + * Retrieve the value associated with a particular key from a (parsed) configuration file + * @param key identifies the value to return + * @return the value + */ + public String getValue(String key) { + return keyValue.get(key); + } + + /** + * Parse a configuration file + * @param inFile is the path to the file + * @throws IOException if the file cannot be read + */ + public void scanConfig(File inFile) throws IOException { + BufferedReader reader = new BufferedReader(new FileReader(inFile)); + + String line; + ConfigLine parse = new ConfigLine(); + + try { + for (;;) { + line = reader.readLine(); + if (line == null) { + break; // End of file reached + } + if (line.length() != 0) { + parse.parseUptoKey(line); + if (parse.key == null) { + continue; + } + String curval = keyValue.get(parse.key); // Check if this is a key we want to find + if (curval != null) { // If this line is setting a value we control + if (curval.length() != 0) { + throw new IOException("Multiple settings for: " + parse.key); + } + parse.parseValue(line); // Discard the original value, but preserve any comment + if (parse.status == 1) { // We have uncommented controlled key + keyValue.put(parse.key, parse.value); + } + } + } + } + } + finally { + reader.close(); + } + } + + /** + * Read in all the entries of the connection file + * @param inFile the file to read in + * @throws IOException if the file cannot be read/parsed + */ + public void scanConnect(File inFile) throws IOException { + BufferedReader reader = new BufferedReader(new FileReader(inFile)); + try { + for (;;) { + String line = reader.readLine(); + if (line == null) { + break; // End of file reached + } + String stripLine = stripConnectComment(line); + if (stripLine == null) { // This line only contained a comment + continue; + } + ConnectLine connLine = new ConnectLine(); + connLine.parse(stripLine); + connectSet.add(connLine); + } + } + finally { + reader.close(); + } + } + + public String getLocalAuthentication() { + for (ConnectLine connLine : connectSet) { + if (connLine.isLocal()) { + return connLine.method; + } + } + return null; + } + + public void setLocalAuthentication(String val, String options) { + for (ConnectLine connLine : connectSet) { + if (connLine.isLocal()) { + connLine.method = val; + connLine.options = options; + } + } + } + + public String getHostAuthentication() { + for (ConnectLine connLine : connectSet) { + if (connLine.type.equals("hostssl") && !connLine.isLocal()) { + return connLine.method; + } + } + return null; + } + + public void setHostAuthentication(String val, String options) { + for (ConnectLine connLine : connectSet) { + if (connLine.type.equals("hostssl") && !connLine.isLocal()) { + connLine.method = val; + connLine.options = options; + } + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/AbstractSQLFunctionDatabase.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/AbstractSQLFunctionDatabase.java new file mode 100644 index 0000000000..a143d806d4 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/AbstractSQLFunctionDatabase.java @@ -0,0 +1,2379 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.sql.*; +import java.util.*; +import java.util.Date; + +import org.apache.commons.lang3.StringUtils; + +import generic.lsh.vector.*; +import ghidra.features.bsim.gui.filters.FunctionTagBSimFilterType; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.client.tables.*; +import ghidra.features.bsim.query.client.tables.CallgraphTable.CallgraphRow; +import ghidra.features.bsim.query.client.tables.DescriptionTable.DescriptionRow; +import ghidra.features.bsim.query.client.tables.ExeTable.ExeTableOrderColumn; +import ghidra.features.bsim.query.client.tables.ExeTable.ExecutableRow; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.*; +import ghidra.util.Msg; + +/** + * Defines the BSim {@link FunctionDatabase} backed by an SQL database. + * + * Simple, one-column tables that only contain string data use the + * {@link SQLStringTable} class and are defined in this class. More complex + * tables are defined in their own classes in the + * {@link ghidra.features.bsim.query.client.tables} package. + * + * @param {@link LSHVectorFactory vector factory} implementation class + */ +public abstract class AbstractSQLFunctionDatabase + implements SQLFunctionDatabase { + + public static final String SQL_TIME_FORMAT = "YYYY-MM-DD HH24:MI:SS.MSz"; + public static final String JAVA_TIME_FORMAT = "yyyy-MM-dd HH:mm:ss.SSSZ"; + + //private static final + + private static final String ARCH_TABLE_NAME = "archtable"; + private static final String COMPILER_TABLE_NAME = "comptable"; + private static final String REPOSITORY_TABLE_NAME = "repotable"; + private static final String PATH_TABLE_NAME = "pathtable"; + private static final String CAT_STRING_TABLE_NAME = "catstringtable"; + + protected final BSimJDBCDataSource ds; + + // Indicates the version of the db table configuration. This needs to be updated + // whenever changes are made to the table structure. A value less-than 0 is ignored + // and disables check (e.g., transient in-memory database). + public final int supportedLayoutVersion; + + /** + * Main connection to database established by call to {@link #initConnection()} + */ + private Connection db = null; + + /** + * Database Information established by {@link #createDatabase(Configuration)} or + * {@link #initializeDatabase(Configuration)} + */ + private DatabaseInformation info; + + // Define all the db tables. + private ExeTable exeTable; + private ExeToCategoryTable exeCategoryTable; + private DescriptionTable descTable; + private CallgraphTable callgraphTable; + + private SQLStringTable archtable; + private SQLStringTable compilertable; + private SQLStringTable repositorytable; + private SQLStringTable pathtable; + private SQLStringTable catstringtable; + + private IdfLookupTable idfLookupTable; + private WeightTable weightTable; + + protected final VF vectorFactory; // Factory used to generate LSHVector objects + + private Error lasterror; + private Status status; + private boolean isinit; + + private KeyValueTable keyValueTable; // used for storing DatabaseInformation + + private OptionalTable[] optionaltables = null; + + private boolean trackcallgraph = true; + + protected AbstractSQLFunctionDatabase(BSimJDBCDataSource ds, VF vectorFactory, + int supportedLayoutVersion) { + this.ds = ds; + this.supportedLayoutVersion = supportedLayoutVersion; + this.vectorFactory = vectorFactory; + + archtable = new SQLStringTable(ARCH_TABLE_NAME, 1000); + compilertable = new SQLStringTable(COMPILER_TABLE_NAME, 1000); + repositorytable = new SQLStringTable(REPOSITORY_TABLE_NAME, 1000); + pathtable = new SQLStringTable(PATH_TABLE_NAME, 1000); + catstringtable = new SQLStringTable(CAT_STRING_TABLE_NAME, 1000); + + callgraphTable = new CallgraphTable(); + exeCategoryTable = new ExeToCategoryTable(catstringtable); + keyValueTable = new KeyValueTable(); + exeTable = + new ExeTable(archtable, compilertable, repositorytable, pathtable, exeCategoryTable); + descTable = new DescriptionTable(exeTable); + + weightTable = new WeightTable(); + idfLookupTable = new IdfLookupTable(); + + lasterror = null; + info = null; + status = Status.Unconnected; + isinit = false; + } + + @Override + public void close() { + + weightTable.close(); + idfLookupTable.close(); + + archtable.close(); + compilertable.close(); + repositorytable.close(); + pathtable.close(); + catstringtable.close(); + + callgraphTable.close(); + exeCategoryTable.close(); + keyValueTable.close(); + exeTable.close(); + descTable.close(); + + if (optionaltables != null) { + for (OptionalTable table : optionaltables) { + table.close(); + } + optionaltables = null; + } + + if (db != null) { + closing(db); + try { + db.close(); + } + catch (SQLException e) { + e.printStackTrace(); + } + db = null; + } + status = Status.Unconnected; + + isinit = false; + info = null; + } + + /** + * Perform any required cleanup prior to connection close. + * Partial cleanup has already been performed. + * @param c database connection + */ + protected void closing(Connection c) { + // do nothing by default + } + + /** + * Establish PostgreSQL DB pooled {@link Connection} for this instance. + * @return database connection + * @throws SQLException if error occurs obtaining connection + */ + protected Connection initConnection() throws SQLException { + if (db == null) { + db = ds.getConnection(); + } + return db; + } + + /** + * Start a transaction which will not be auto-committed. The {@link #endTransaction(boolean)} + * method must be invoked when the transaction has completed or failed and should be performed + * within a finally block. + * @param lockTablesForWrite if true a table lock will be placed which will clear when + * {@link #endTransaction(boolean)} is invoked. + * @return the connection associated with the transaction + * @throws SQLException if an error occurs + */ + protected Connection beginTransaction(boolean lockTablesForWrite) throws SQLException { + db.setAutoCommit(false); + if (lockTablesForWrite) { + lockTablesForWrite(); + } + return db; + } + + /** + * Lock appropriate tables during complex write/update operation + * @throws SQLException if an error occurs + */ + protected void lockTablesForWrite() throws SQLException { + // do nothing + } + + /** + * End a transaction which was started with {@link #beginTransaction(boolean)} and return + * DB connection to an auto-commit state. + * @param commit true if transaction should be commited, false to force a rollback. + * @throws SQLException if an error occurs + */ + protected void endTransaction(boolean commit) throws SQLException { + if (commit) { + db.commit(); + } + else { + db.rollback(); + } + db.setAutoCommit(true); + } + + /** + * @param erec the exectuable record representing the exec to delete + * @param funclist list of functions to remove from the callgraph table + * @param hasCategories if true, deletes entries from the category table + * @return the number of rows deleted + * @throws SQLException if there's an error deleting rows + */ + private int deleteExecutable(ExecutableRecord erec, List funclist, + boolean hasCategories) throws SQLException { + + if (hasCategories) { + exeCategoryTable.delete(erec.getRowId().getLong()); + } + if (trackcallgraph) { + for (FunctionDescription element : funclist) { + callgraphTable.delete(element.getId().getLong()); + } + } + return deleteExeRows(erec); + } + + /** + * + * @param erec the executable record to delete + * @return the number of rows deleted + * @throws SQLException if there is an error removing from the + */ + private int deleteExeRows(ExecutableRecord erec) throws SQLException { + + long rowid = erec.getRowId().getLong(); + final int delcount = descTable.delete(rowid); + exeTable.delete(rowid); // don't need to save the return value + return delcount; + } + + /** + * + * @param dbIndo the database information object + * @throws SQLException if there is an error parsing the key/value table + */ + private void readExecutableCategories(DatabaseInformation dbIndo) throws SQLException { + int count = Integer.parseInt(keyValueTable.getValue("execatcount")); + if (count <= 0) { + dbIndo.execats = null; + return; + } + dbIndo.execats = new ArrayList(); + for (int i = 0; i < count; ++i) { + String key = "execat" + Integer.toString(i + 1); + String value = keyValueTable.getValue(key); + dbIndo.execats.add(value); + } + } + + /** + * @param dbInfo the database information object + * @throws SQLException if there is an error parsing the key/value table + */ + private void readFunctionTags(DatabaseInformation dbInfo) throws SQLException { + String countString = keyValueTable.getValue("functiontagcount"); + int count; + if (countString != null) { + count = Integer.parseInt(countString); + } + else { + count = 0; // key may not be present, assume 0 tags + } + if (count <= 0) { + dbInfo.functionTags = null; + return; + } + dbInfo.functionTags = new ArrayList(); + for (int i = 0; i < count; ++i) { + String key = "functiontag" + (i + 1); + String value = keyValueTable.getValue(key); + dbInfo.functionTags.add(value); + } + } + + /** + * Parse keyValueTable data into a {@link DatabaseInformation} object. + * @return a database information object + * @throws SQLException if there is an error parsing the key/value table + */ + private DatabaseInformation parseDatabaseInfo() throws SQLException { + DatabaseInformation dbInfo = new DatabaseInformation(); + dbInfo.databasename = keyValueTable.getValue("name"); + dbInfo.owner = keyValueTable.getValue("owner"); + dbInfo.description = keyValueTable.getValue("description"); + dbInfo.major = (short) Integer.parseInt(keyValueTable.getValue("major")); + dbInfo.minor = (short) Integer.parseInt(keyValueTable.getValue("minor")); + dbInfo.settings = Integer.parseInt(keyValueTable.getValue("settings")); + String val = keyValueTable.getValue("readonly"); + dbInfo.readonly = false; + if (val.length() > 0) { + dbInfo.readonly = (val.charAt(0) == 't'); + } + val = keyValueTable.getValue("trackcallgraph"); + dbInfo.trackcallgraph = false; + if (val.length() > 0) { + dbInfo.trackcallgraph = (val.charAt(0) == 't'); + } + try { + dbInfo.layout_version = Integer.parseInt(keyValueTable.getValue("layout")); + } + catch (SQLException exception) { // If this is an old layout, it may + // not have this key + dbInfo.layout_version = 0; // In which case, we know it is version 0 + } + dbInfo.dateColumnName = keyValueTable.getValue("datecolumn"); + if (dbInfo.dateColumnName.equals("Ingest Date")) { + // name + dbInfo.dateColumnName = null; // Don't bother holding it + } + readExecutableCategories(dbInfo); + readFunctionTags(dbInfo); + return dbInfo; + } + + /** + * Initialize table state and {@code config} from existing database. + * NOTE: it is left to specific implementation to override this method to properly + * initialize default {@code config.weightfactory } and {@code config.idflookup }. + * @param config the database configuration + * @throws SQLException if there is a problem opening the database connection + */ + protected void initializeDatabase(Configuration config) throws SQLException { + + initConnection(); + setConnectionOnTables(db); + + try (Statement st = db.createStatement()) { + + config.info = parseDatabaseInfo(); + config.k = Integer.parseInt(keyValueTable.getValue("k")); + config.L = Integer.parseInt(keyValueTable.getValue("L")); + config.weightfactory = new WeightFactory(); + config.idflookup = new IDFLookup(); + + trackcallgraph = config.info.trackcallgraph; + } + + weightTable.recoverWeights(config.weightfactory); + idfLookupTable.recoverIDFLookup(config.idflookup); + } + + /** + * Generate new empty database which corresponds to {@link #ds}. + * A new connection may be used to the default database + * so that a new database may be created. + * @throws SQLException if an error occurs while creating the database + */ + protected abstract void generateRawDatabase() throws SQLException; + + /** + * Generate tables and initialize state for a new database. + * @param config the database configuration + * @throws SQLException if there is aproblem opening a connectino to the database + */ + protected void createDatabase(Configuration config) throws SQLException { + generateRawDatabase(); + trackcallgraph = config.info.trackcallgraph; + + initConnection(); + setConnectionOnTables(db); + + try (Statement st = db.createStatement()) { + beginTransaction(false); + boolean commit = false; + try { + archtable.createTable(); + compilertable.createTable(); + repositorytable.createTable(); + pathtable.createTable(); + catstringtable.createTable(); + + keyValueTable.create(st); + keyValueTable.writeBasicInfo(config.info); + keyValueTable.insert("k", Integer.toString(config.k)); + keyValueTable.insert("L", Integer.toString(config.L)); + commit = true; + } + finally { + endTransaction(commit); + } + + exeCategoryTable.create(st); + exeTable.create(st); + descTable.create(st); + + if (trackcallgraph) { + callgraphTable.create(st); + } + + installWeights(db, config.weightfactory); + installIDFLookup(db, config.idflookup); + } + catch (final SQLException err) { + throw new SQLException("Could not create database: " + err.getMessage()); + } + } + + protected void setConnectionOnTables(Connection db) { + + weightTable.setConnection(db); + idfLookupTable.setConnection(db); + + archtable.setConnection(db); + compilertable.setConnection(db); + repositorytable.setConnection(db); + pathtable.setConnection(db); + catstringtable.setConnection(db); + + callgraphTable.setConnection(db); + exeCategoryTable.setConnection(db); + keyValueTable.setConnection(db); + exeTable.setConnection(db); + descTable.setConnection(db); + } + + /** + * Installs/Replaces Weight Table + * @param factory the weight factory + * @throws SQLException if there is an error creating the database query + */ + private void installWeights(Connection conn, WeightFactory factory) throws SQLException { + try (Statement st = conn.createStatement()) { + weightTable.drop(st); + weightTable.create(st); + + double[] weightArray = factory.toArray(); + beginTransaction(false); + boolean commit = false; + try { + for (int i = 0; i < weightArray.length; ++i) { + weightTable.insert(i, weightArray[i]); + } + commit = true; + } + finally { + endTransaction(commit); + } + } + } + + /** + * Installs/Replaces the IDF Lookup table + * @param lookup the IDF lookup + * @throws SQLException if there is an error creating the database query + */ + private void installIDFLookup(Connection conn, IDFLookup lookup) throws SQLException { + try (Statement st = conn.createStatement()) { + idfLookupTable.drop(st); + idfLookupTable.create(st); + + int[] intArray = lookup.toArray(); + beginTransaction(false); + boolean commit = false; + try { + for (int i = 0; i < intArray.length; i += 2) { + idfLookupTable.insert(intArray[i + 1], intArray[i]); + } + commit = true; + } + finally { + endTransaction(commit); + } + } + } + + /** + * + * @param rec the function description update object + * @throws SQLException if there is a problem creating the query statement + */ + private void updateFunction(FunctionDescription.Update rec) throws SQLException { + try (Statement st = db.createStatement()) { + StringBuilder buf = new StringBuilder(); + buf.append("UPDATE desctable SET "); + boolean previous = false; + if (rec.function_name) { + previous = true; + buf.append("name_func='"); + appendEscapedLiteral(buf, rec.update.getFunctionName()); + buf.append('\''); + } + if (rec.flags) { + if (previous) { + buf.append(','); + } + buf.append("flags="); + buf.append(rec.update.getFlags()); + } + buf.append(" WHERE id = "); + buf.append(rec.update.getId().getLong()); + + st.executeUpdate(buf.toString()); + } + } + + /** + * Make a temporary ExecutableRecord given database row information. The + * record is NOT attached to a DescriptionManager + * + * @param row + * is the columnar values for the executable from the database + * @return the new temporary ExecutableRecord + * @throws SQLException if there is a problem parsing the table objects + */ + private ExecutableRecord makeExecutableRecordTemp(ExecutableRow row) throws SQLException { + String arch = archtable.getString(row.arch_id); + ExecutableRecord exeres; + RowKeySQL rowid = new RowKeySQL(row.rowid); + if (ExecutableRecord.isLibraryHash(row.md5)) { + exeres = new ExecutableRecord(row.exename, arch, rowid); + } + else { + String cname = compilertable.getString(row.compiler_id); + String repo = repositorytable.getString(row.repo_id); + String path = null; + if (repo != null) { + path = pathtable.getString(row.path_id); + } + exeres = new ExecutableRecord(row.md5, row.exename, cname, arch, + new Date(row.date_milli), rowid, repo, path); + } + return exeres; + } + + /** + * Run through a list of functions: mark any that have been previously + * stored to the database by populating the function's rowid in the + * corresponding FunctionDescription + * + * @param input + * is DescriptionManager containing the functions + * @param iter + * is the iterator listing the FunctionDescriptions + * @return true if there are ANY new functions, either from new executables, + * or old executables with new functions + * @throws SQLException if there is a problem querying the description table + */ + private boolean markPreviouslyStoredFunctions(DescriptionManager input, + Iterator iter) throws SQLException { + boolean newfuncs = false; + + while (iter.hasNext()) { + FunctionDescription func = iter.next(); + ExecutableRecord erec = func.getExecutableRecord(); + if (!erec.isAlreadyStored()) { + newfuncs = true; + continue; + } + DescriptionRow row = descTable.queryFuncNameAddr(erec.getRowId().getLong(), + func.getFunctionName(), func.getAddress()); + if (row == null) { + newfuncs = true; + continue; + } + + // We could do some consistency checks here between -func- being + // inserted and -func2- that was already present + + // Set the id, which marks -func- as already stored. + input.setFunctionDescriptionId(func, new RowKeySQL(row.rowid)); + } + return newfuncs; + } + + /** + * + * @param value the architecture value + * @return the architecture ID + * @throws SQLException if there is a problem parsing the architecture value + */ + long queryArchString(String value) throws SQLException { + return archtable.readStringId(value); + } + + /** + * + * @param value the compiler value + * @return the compiler ID + * @throws SQLException if there is a problem parsing the compiler value + */ + long queryCompilerString(String value) throws SQLException { + return compilertable.readStringId(value); + } + + /** + * + * @param value the repository value + * @return the repository ID + * @throws SQLException if there is a problem parsing the repository value + */ + long queryRepositoryString(String value) throws SQLException { + return repositorytable.readStringId(value); + } + + /** + * @param value the category value + * @return the category ID + * @throws SQLException if there is a problem parsing the category value + */ + long queryCategoryString(String value) throws SQLException { + return catstringtable.readStringId(value); + } + + /** + * Query for all functions (up to the limit -max-) of the given -exe-. + * Populate DescriptionManager and List and with corresponding FunctionDescription objects + * Does NOT populate SignatureRecord or CallGraph parts of the FunctionDescription + * @param vecres has all queried functions added to it + * @param exe is the executable containing the functions + * @param res is the DescriptionManager container for the new records + * @param max is the maximum number of records to read + * @return the number of function records read + * @throws SQLException if there is a problem creating or executing the query + */ + private int queryAllFunc(List vecres, ExecutableRecord exe, + DescriptionManager res, int max) throws SQLException { + StringBuffer buf = new StringBuffer(); + buf.append("SELECT ALL * FROM desctable WHERE id_exe = "); + buf.append((int) exe.getRowId().getLong()); + if (max > 0) { + buf.append(" LIMIT ").append(max); + } + try (Statement st = db.createStatement(); ResultSet rs = st.executeQuery(buf.toString())) { + st.setFetchSize(50); + int count = 0; + final List descrow = descTable.extractDescriptionRows(rs, max); + count = descrow.size(); + descTable.convertDescriptionRows(vecres, descrow, exe, res, null); + return count; + } + } + + /** + * Update metadata for the executable -erec- and all its functions (in + * manage) + * + * @param manage + * is the collection of functions to update + * @param erec + * is the root executable + * @param badfunc + * collects references to functions with update info that could + * not be identified + * @param checkExeCategories if true, update any found executable categories + * @return -1 if the executable could not be found, otherwise return 2*# of + * update functions + 1 if the executable metadata is also updated + * @throws LSHException if there is a problem transferring the executable to a new container + * @throws SQLException if there is a problem executing various queries + */ + private int updateExecutable(DescriptionManager manage, ExecutableRecord erec, + List badfunc, boolean checkExeCategories) + throws LSHException, SQLException { + int val; // The return value + beginTransaction(true); + boolean commit = false; + try { + ExecutableRow row = exeTable.queryMd5ExeMatch(erec.getMd5()); + if (row == null) { + return -1; // Indicate that we couldn't find the executable + } + ExecutableRecord erec_db = makeExecutableRecordTemp(row); + DescriptionManager dbmanage = new DescriptionManager(); + erec_db = dbmanage.transferExecutable(erec_db); + if (checkExeCategories) { + final List catvec = + exeCategoryTable.queryExecutableCategories(erec_db.getRowId().getLong(), 100); // Make sure categories are filled in before diff + + dbmanage.setExeCategories(erec_db, catvec); + } + ExecutableRecord.Update exe_update = new ExecutableRecord.Update(); + boolean has_exe_update = erec.diffForUpdate(exe_update, erec_db); + + // Load all the functions in the database under this executable + List funclist = new ArrayList(); + queryAllFunc(funclist, erec_db, dbmanage, 0); + + // Create a map from address to executables + Map addrmap = + FunctionDescription.createAddressToFunctionMap(funclist.iterator()); + + // Match new functions to old functions via the address + List updatelist; + updatelist = + FunctionDescription.generateUpdates(manage.listFunctions(erec), addrmap, badfunc); + + if (!has_exe_update && updatelist.isEmpty()) { + return 0; // All updates are in place already + } + + // Do the actual database updates + if (has_exe_update) { + exeTable.updateExecutable(exe_update); + } + for (FunctionDescription.Update updateRec : updatelist) { + updateFunction(updateRec); + } + commit = true; + val = has_exe_update ? 1 : 0; + val += 2 * updatelist.size(); + } + finally { + endTransaction(commit); + } + return val; + } + + /** + * Establish an optional key/value table with this connection. + * The OptionalTable object is created and added to the list for this connection. + * If the caller desires, the table is tested for existence. If it doesn't + * exist, the object is not added to the list and null is returned. + * @param tableName is the name of the SQL table + * @param keyType is the type-code of the key column + * @param valueType is the type-code of the value column + * @param testExistence if true, we test the existence of the table + * @return the OptionalTable or null + * @throws SQLException for problems with the connection, or if + * the table exists with different column types + */ + private OptionalTable getOptionalTable(String tableName, int keyType, int valueType, + boolean testExistence) throws SQLException { + if (optionaltables != null) { // Search for existing table + for (OptionalTable table : optionaltables) { + if (table.getName().equals(tableName)) { + if (keyType != table.getKeyType() || valueType != table.getValueType()) { + throw new SQLException("Optional table: column type mismatch"); + } + return table; + } + } + } + // If we reach here, table object doesn't exist, so we create it + OptionalTable table = new OptionalTable(tableName, keyType, valueType, db); + if (testExistence) { // If the user requested + if (!table.exists()) { // test for the existence of the table + table.close(); + return null; // If it doesn't exist, don't save new table object, return null + } + } + // Insert the new table object at the end of the list + OptionalTable[] newArray; + if (optionaltables != null) { + newArray = Arrays.copyOf(optionaltables, optionaltables.length + 1); + newArray[optionaltables.length] = table; + } + else { + newArray = new OptionalTable[1]; + newArray[0] = table; + } + optionaltables = newArray; + return table; + } + + /** + * + * @param buf the string builder object + * @param str the string to parse + * @throws SQLException if there is a zero byte in the string + */ + public static void appendEscapedLiteral(StringBuilder buf, String str) throws SQLException { + for (int i = 0; i < str.length(); ++i) { + char ch = str.charAt(i); + if (ch == '\0') { + throw new SQLException("Zero byte in SQL string"); + } + if (ch == '\\' || ch == '\'') { + buf.append(ch); + } + buf.append(ch); + } + } + + /** + * Convert a low-level list of function rows into full FunctionDescription objects + * @param simres (optional -- may be null) generate a SimilarityNote for every function + * @param descvec is the list of low-level function rows + * @param vecres (optional -- may be null) vector result producing these functions + * @param res is the DescriptionManager holding the newly generated FunctionDescriptions + * @param srec (optional -- may be null) is a description object of the vector + * @throws SQLException is there is an error querying tables + * @throws LSHException for internal consistency errors + */ + protected void convertDescriptionRows(SimilarityResult simres, List descvec, + VectorResult vecres, DescriptionManager res, SignatureRecord srec) + throws SQLException, LSHException { + Iterator iter = descvec.iterator(); + DescriptionRow currow = iter.next(); + RowKey rowKey = new RowKeySQL(currow.id_exe); + ExecutableRecord curexe = res.findExecutableByRow(rowKey); + if (curexe == null) { + ExecutableRow exerow = exeTable.querySingleExecutableId(currow.id_exe); + curexe = exeTable.makeExecutableRecord(res, exerow); + res.cacheExecutableByRow(curexe, rowKey); + } + + FunctionDescription fres = + DescriptionTable.convertDescriptionRow(currow, curexe, res, srec); + if (simres != null) { + simres.addNote(fres, vecres.sim, vecres.signif); + } + if (srec != null) { + res.setSignatureId(srec, currow.id_sig); + } + while (iter.hasNext()) { + currow = iter.next(); + rowKey = new RowKeySQL(currow.id_exe); + curexe = res.findExecutableByRow(rowKey); + if (curexe == null) { + ExecutableRow exerow = exeTable.querySingleExecutableId(currow.id_exe); + curexe = exeTable.makeExecutableRecord(res, exerow); + res.cacheExecutableByRow(curexe, rowKey); + } + fres = DescriptionTable.convertDescriptionRow(currow, curexe, res, srec); + if (simres != null) { + simres.addNote(fres, vecres.sim, vecres.signif); + } + } + } + + /** + * For the given -func-, query the database for its children and put their FunctionDescriptions -res- + * @param func is the parent FunctionDescription + * @param manager is the container DescriptionManager + * @param funcmap quick lookup of FunctionDescription objects already in -res- + * @throws SQLException if there is a problem querying callgraph table + * @throws LSHException if there is a problem querying the description table + */ + private void fillinChildren(FunctionDescription func, DescriptionManager manager, + Map funcmap) throws SQLException, LSHException { + List callvec = callgraphTable.queryCallgraphRows(func, trackcallgraph); + for (CallgraphRow element : callvec) { + long id = element.dest; + RowKey key = new RowKeySQL(id); + FunctionDescription fdesc = funcmap.get(key); + if (fdesc == null) { + fdesc = descTable.querySingleDescriptionId(manager, id); + funcmap.put(fdesc.getId(), fdesc); + } + manager.makeCallgraphLink(func, fdesc, 0); + } + } + + /** + * + * @param manage the description manager + * @throws SQLException if there is a problem querying for executable categories + */ + private void fillinExecutableCategories(DescriptionManager manage) throws SQLException { + TreeSet exes = manage.getExecutableRecordSet(); + int max = 100; // Max number of categories that can be returned for a single executable + // TODO: Find better way to manage this and error out if limit exceeded + for (ExecutableRecord erec : exes) { + if (erec.categoriesAreSet()) { + continue; // Categories are already filled in + } + List catvec = + exeCategoryTable.queryExecutableCategories(erec.getRowId().getLong(), max); + manage.setExeCategories(erec, catvec); + } + } + + /** + * + * @param iter the histogram id iterator + * @throws SQLException if there is a problem deleting vectors + */ + private void deleteVectors(Iterator iter) throws SQLException { + while (iter.hasNext()) { + IdHistogram hist = iter.next(); + deleteVectors(hist.id, hist.count); + } + } + + /** + * Low level count decrement of a vector record from vectable, if count + * reaches zero, the record is deleted + * @param id vector row ID + * @param countdiff the amount to subtract from count + * @return 0 if decrement short of 0, return 1 if record was removed, return + * -1 if there was a problem + * @throws SQLException if there is a problem creating or executing the query + */ + protected abstract int deleteVectors(long id, int countdiff) throws SQLException; + + long recoverExternalFunctionId(String exename, String functionname, String reparch) + throws SQLException, LSHException { + String md5 = ExecutableRecord.calcLibraryMd5Placeholder(exename, reparch); + ExecutableRow row = exeTable.queryMd5ExeMatch(md5); + if (row == null) { + throw new LSHException("Could not resolve filter specifying executable: " + exename); + } + + DescriptionRow descRow = descTable.queryFuncNameAddr(row.rowid, functionname, -1); + if (descRow == null) { + throw new LSHException( + "Could not resolve filter specifying function: [" + exename + "]" + functionname); + } + return descRow.rowid; + } + + // Pulled-in from old FunctionDatabaseClient + + /** + * Make sure the FunctionDescription has its attached SignatureRecord, if not, query for it + * @param functionDescription is the FunctionDescription + * @param descriptionManager is the container + * @param sigmap is a container of cached SignatureRecords which is checked before querying (may be null) + * @throws SQLException if there is a problem querying the vector ID + */ + private void queryAssociatedSignature(FunctionDescription functionDescription, + DescriptionManager descriptionManager, Map sigmap) + throws SQLException { + if (functionDescription.getSignatureRecord() != null) { + return; + } + long vectorId = functionDescription.getVectorId(); + if (vectorId == 0) { + return; // We don't know the id + } + VectorResult rowres; + SignatureRecord srec; + if (sigmap != null) { + srec = sigmap.get(vectorId); + if (srec == null) { + rowres = queryVectorId(vectorId); + srec = descriptionManager.newSignature(rowres.vec, rowres.hitcount); + descriptionManager.setSignatureId(srec, vectorId); + sigmap.put(vectorId, srec); + } + } + else { + rowres = queryVectorId(vectorId); + srec = descriptionManager.newSignature(rowres.vec, rowres.hitcount); + descriptionManager.setSignatureId(srec, vectorId); + } + descriptionManager.attachSignature(functionDescription, srec); + } + + /** + * + * @param id the vector id + * @return the vector result + * @throws SQLException if there is a problem creating or executing the query + */ + protected abstract VectorResult queryVectorId(long id) throws SQLException; + + /** + * Make sure every FunctionDescription object has its associated SignatureRecord + * Read the SignatureRecords from the database if necessary + * + * @param funcvec list of functions + * @param manager container DescriptionManager + * @throws SQLException if there is a problem querying the signature + */ + private void queryAssociatedSignatures(List funcvec, + DescriptionManager manager) throws SQLException { + TreeMap sigmap = new TreeMap<>(); + for (FunctionDescription element : funcvec) { + queryAssociatedSignature(element, manager, sigmap); + } + } + + /** + * Returns the total number of hits in the given result set. + * + * @param resultSet the list of vector results + * @return the number of hits + */ + private int getTotalCount(List resultSet) { + int count = 0; + for (VectorResult res : resultSet) { + count += res.hitcount; + } + return count; + } + + /** + * This is an internal function for the -insert- capability + * It runs through executables in -input- and tests if similar records are already present + * If an executable existed in the database previously: + *
    + *
  • if the executable is NOT a library it throws DatabaseNonFatalException to indicate this file is already ingested + *
  • if the executable is a library then + *
      + *
    • it runs through all functions associated with the library + *
    • if there is already a record for the function it marks the object in -input- so + * that a duplicate won't get entered for that function. + *
    + *
+ * @param input is the DescriptionManager + * @throws LSHException if this executable has already been inserted + * @throws SQLException if there is an error issuing the query + * @throws DatabaseNonFatalException if a library has already been ingested (non-fatal) + */ + private void testExecutableDuplication(DescriptionManager input) + throws SQLException, LSHException, DatabaseNonFatalException { + boolean pickout_storedfuncs = false; + for (ExecutableRecord erec : input.getExecutableRecordSet()) { + ExecutableRow row = exeTable.queryMd5ExeMatch(erec.getMd5()); + if (row != null) { // Already have a matching executable + ExecutableRecord tmp = makeExecutableRecordTemp(row); + int cmp = tmp.compareMetadata(erec); + if (cmp != 0) { + String fatalerror = FunctionDatabase.constructFatalError(cmp, erec, tmp); + if (fatalerror != null) { + throw new LSHException(fatalerror); + } + throw new DatabaseNonFatalException( + FunctionDatabase.constructNonfatalError(cmp, erec, tmp)); + } + if (erec.getRowId() != null) { + if (!erec.getRowId().equals(tmp.getRowId())) { + throw new LSHException( + "Id mismatch when inserting executable: " + erec.getNameExec()); + } + } + else { + input.setExeRowId(erec, tmp.getRowId()); + } + input.setExeAlreadyStored(erec); + // Just because we've seen a library before doesn't mean we should stop function insertion + // Libraries are likely to be partially inserted multiple times + if (!erec.isLibrary()) { + throw new DatabaseNonFatalException( + erec.getNameExec() + " is already ingested"); + } + pickout_storedfuncs = true; // At least one executable has previously inserted functions + } + } + + if (pickout_storedfuncs) { + if (!markPreviouslyStoredFunctions(input, input.listAllFunctions())) { + throw new DatabaseNonFatalException("Already inserted"); + } + } + } + + /** + * Do the final work of inserting new ExecutableRecords into the database. This function + * assumes testExecutableDuplication has already run and marked previously ingested records + * + * @param input the executable descriptor + * @throws SQLException if database records cannot be inserted + */ + private void commitExecutables(DescriptionManager input) throws SQLException { + Iterator iter = input.getExecutableRecordSet().iterator(); + while (iter.hasNext()) { + ExecutableRecord erec = iter.next(); + if (erec.isAlreadyStored()) { + continue; + } + long newid = exeTable.insert(erec); + input.setExeRowId(erec, new RowKeySQL(newid)); + } + if (info.execats != null) { + iter = input.getExecutableRecordSet().iterator(); + while (iter.hasNext()) { + ExecutableRecord erec = iter.next(); + exeCategoryTable.storeExecutableCategories(erec); + } + } + } + + @Override + public Status getStatus() { + if (status == Status.Unconnected) { + // defer to data source for connection pool status + return ds.getStatus(); + } + return status; + } + + @Override + public ConnectionType getConnectionType() { + return ds.getConnectionType(); + } + + @Override + public String getUserName() { + return null; + } + + @Override + public void setUserName(String userName) { + // ignore + } + + @Override + public LSHVectorFactory getLSHVectorFactory() { + return vectorFactory; + } + + @Override + public DatabaseInformation getInfo() { + return info; + } + + @Override + public int compareLayout() { + // version ignored if supportedLayoutVersion < 0 + if (supportedLayoutVersion < 0 || info.layout_version == supportedLayoutVersion) { + return 0; + } + return (info.layout_version < supportedLayoutVersion) ? -1 : 1; + } + + @Override + public BSimServerInfo getServerInfo() { + return ds.getServerInfo(); + } + + @Override + public String getURLString() { + return ds.getServerInfo().toURLString(); + } + + @Override + public Error getLastError() { + return lasterror; + } + + /** + * Create a new database + * + * @param config is the configuration information for the database + * @throws SQLException if a database connection cannot be established + */ + private void generate(Configuration config) throws SQLException { + config.info.layout_version = supportedLayoutVersion; + info = config.info; + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + + createDatabase(config); + status = Status.Ready; + isinit = true; + } + + @Override + public boolean initialize() { + if (isinit) { + return true; + } + try { + Configuration config = new Configuration(); + initializeDatabase(config); + info = config.info; + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + } + catch (CancelledSQLException e) { + status = Status.Error; + lasterror = new Error(ErrorCategory.AuthenticationCancelled, + "Authentication cancelled by user"); + return false; + } + catch (SQLException err) { + status = Status.Error; + // Pooled connections add an additional wrapper + Throwable cause = err.getCause(); + if (cause == null) { + cause = err; + } + String msg = cause.getMessage(); + if (msg.contains("already in use:")) { + lasterror = new Error(ErrorCategory.Initialization, + "Database already in use by another process"); + } + else if (msg.contains("authentication failed") || + msg.contains("requires a valid client certificate")) { + lasterror = + new Error(ErrorCategory.Authentication, "Could not authenticate with database"); + } + else if (msg.contains("does not exist") && !msg.contains(" role ")) { + lasterror = new Error(ErrorCategory.Nodatabase, cause.getMessage()); + } + else { + lasterror = new Error(ErrorCategory.Initialization, + "Database error on initialization: " + cause.getMessage()); + } + return false; + } +// catch (NoDatabaseException err) { +// info = null; +// lasterror = new Error(ErrorCategory.Nodatabase, +// "Database has not been created yet: " + err.getMessage()); +// +// isinit = true; +// status = Status.Ready; +// return true; +// } + + status = Status.Ready; + isinit = true; + return true; + } + + /** + * Insert a set of (possible callgraph interrelated) functions into database, + * @param input is the DescriptionManager container of the functions and executables + * + * @throws LSHException if there are duplicate executables + * @throws SQLException if there is an error issuing the query + * @throws DatabaseNonFatalException if there are duplicate executables + */ + private void insert(DescriptionManager input) + throws SQLException, LSHException, DatabaseNonFatalException { + + beginTransaction(true); + boolean commit = false; + try { + testExecutableDuplication(input); + commitExecutables(input); + Iterator iter = input.listAllFunctions(); + long start_id = 0; + while (iter.hasNext()) { + FunctionDescription func = iter.next(); + if (func.getId() != null) { + continue; // Already inserted + } + SignatureRecord srec = func.getSignatureRecord(); + if (srec != null) { + long sig_id = storeSignatureRecord(srec); + input.setSignatureId(srec, sig_id); + } + long id = descTable.insert(func); + if (start_id == 0) { + start_id = id; + } + input.setFunctionDescriptionId(func, new RowKeySQL(id)); + } + if (trackcallgraph) { + iter = input.listAllFunctions(); + while (iter.hasNext()) { + FunctionDescription func = iter.next(); + if (func.getId().getLong() < start_id) { + continue; // Already inserted + } + callgraphTable.insert(func); + } + } + commit = true; + } + finally { + endTransaction(commit); + } + } + + /** + * Make sure the vector corresponding to the SignatureRecord is inserted into the vectable + * @param sigrec is the SignatureRecord + * @return the computed id of the vector + * @throws SQLException if there is a problem creating or executing the query + */ + protected abstract long storeSignatureRecord(SignatureRecord sigrec) throws SQLException; + + /** + * Query the database for functions with a similar feature vector to -vec- + * @param simres receives the list of results and their similarity to the base vector + * @param res is the DescriptionManager container for the results + * @param vec is the feature vector to match + * @param query contains various thresholds for the query + * @param filter specifies additional conditions functions (and exes) must meet after meeting sim/signif threshold + * @param vecToResultsMap list of result vectors + * @throws SQLException if there is an error issuing the query + * @throws LSHException if executable record creation fails + */ + private void queryNearest(SimilarityResult simres, DescriptionManager res, LSHVector vec, + QueryNearest query, BSimSqlClause filter, + HashMap> vecToResultsMap) + throws SQLException, LSHException { + List resultset = new ArrayList<>(); + int vectormax = query.vectormax; + if (vectormax == 0) { + vectormax = 2000000; // Really means a very big limit + } + + // Check to see if we've already queried for this vector before. If so, just grab + // the results from the map. + if (vecToResultsMap.containsKey(vec)) { + resultset = vecToResultsMap.get(vec); + simres.setTotalCount(getTotalCount(resultset)); + } + else { + // Perform the query. + simres.setTotalCount( + queryNearestVector(resultset, vec, query.thresh, query.signifthresh, vectormax)); + + // Put the new results in the map so we can use them if another + // similar vector comes along. + vecToResultsMap.put(vec, resultset); + } + + int count = 0; + + for (VectorResult dresult : resultset) { + if (count >= query.max) { + break; + } + count += retrieveFuncDescFromVectors(dresult, res, count, query, filter, simres); + } + } + + /** + * Generates {@link FunctionDescription} objects for each function with signature vector + * {@code dresult}. If a non-null {@code filter} is provided, {@link FunctionDescription}s + * are only created for functions with pass the filter. + * + * @param dresult {@link VectorResult} representing a matching vector + * @param res {@link DescriptionManager} container for the results + * @param count number of function descriptions which have already been returned for this base + * vector + * @param query contains various thresholds for the query + * @param filter specifies additional conditions functions (and exes) must meet after + * meeting sim/signif threshold + * @param simres receives the list of results and their similarity to the base vector + * @return number of + * @throws SQLException if there is an error issuing the query + * @throws LSHException if executable record creation fails + */ + protected int retrieveFuncDescFromVectors(VectorResult dresult, DescriptionManager res, + int count, QueryNearest query, BSimSqlClause filter, SimilarityResult simres) + throws SQLException, LSHException { + SignatureRecord srec = res.newSignature(dresult.vec, dresult.hitcount); + List descres; + if (filter == null) { + descres = descTable.queryVectorIdMatch(dresult.vectorid, query.max - count); + } + else { + descres = descTable.queryVectorIdMatchFilter(dresult.vectorid, filter.tableClause(), + filter.whereClause(), query.max - count); + } + if (descres == null) { + throw new SQLException("Error querying vectorid: " + Long.toString(dresult.vectorid)); + } + if (descres.size() == 0) { + if (filter != null) { + return 0; // Filter may have eliminated all results + } + // Otherwise this is a sign of corruption in the database + throw new SQLException( + "No functions matching vectorid: " + Long.toString(dresult.vectorid)); + } + convertDescriptionRows(simres, descres, dresult, res, srec); + return descres.size(); + } + + /** + * + * @param resultset the list of result set objects to populate + * @param vec the vector containing the saveSQL query statement + * @param simthresh the similarity threshold + * @param sigthresh the confidence threshold + * @param max the max number of results to return + * @return the number of results returned + * @throws SQLException if there is a problem creating or executing the query + */ + protected abstract int queryNearestVector(List resultset, LSHVector vec, + double simthresh, double sigthresh, int max) throws SQLException; + + /** + * Perform nearest vector query based upon specified query which contains response + * record which should be filled-in with search results. + * + * @param query the nearest vector query to perform + * @throws SQLException if there is an error issuing the query + */ + protected abstract void queryNearestVector(QueryNearestVector query) throws SQLException; + + /** + * Query for function names within a previously queried executable -erec- + * + * @param functionDescriptions list of functions to be filled in by the query (may be null) + * @param manager container for record + * @param executableRecord previously queried ExecutableRecord + * @param functionName name to query for, if empty string, all functions in executable are returned + * @param shouldReturnSignatures true if signature of queried functions should also be returned + * @param maxResults maximum results to return + * @throws SQLException if there is an error issuing the query + */ + private void queryByName(List functionDescriptions, + DescriptionManager manager, ExecutableRecord executableRecord, String functionName, + boolean shouldReturnSignatures, int maxResults) throws SQLException { + if (functionDescriptions == null) { + functionDescriptions = new ArrayList<>(); + } + if (functionName.length() == 0) { + queryAllFunc(functionDescriptions, executableRecord, manager, maxResults); + } + else { + List descres = descTable + .queryFuncName(executableRecord.getRowId().getLong(), functionName, maxResults); + descTable.convertDescriptionRows(functionDescriptions, descres, executableRecord, + manager, null); + } + if (shouldReturnSignatures) { + queryAssociatedSignatures(functionDescriptions, manager); + } + } + + /** + * Query for single function by name and address, within an executable + * @param manager - container for record + * @param erec - previously queried ExecutableRecord + * @param funcname - name of function to query for + * @param address - address of function to query for + * @param sigs - true if signature of function should also be returned + * @return the resulting FunctionDescription or null + * @throws SQLException if there is an error issuing the query + */ + private FunctionDescription queryByNameAddress(DescriptionManager manager, + ExecutableRecord erec, String funcname, long address, boolean sigs) + throws SQLException { + DescriptionRow row = + descTable.queryFuncNameAddr(erec.getRowId().getLong(), funcname, address); + FunctionDescription func = DescriptionTable.convertDescriptionRow(row, erec, manager, null); + if (sigs) { + queryAssociatedSignature(func, manager, null); + } + return func; + } + + /** + * Returns a list of all executables in the repository. + * + * @param manager the description manager + * @param limit the max number of results to return + * @param filterMd5 md5 filter + * @param filterExeName executable name filter + * @param filterArch architecture filter + * @param filterCompilerName compiler name filter + * @param columnName the primary sort column name + * @param includeFakes if false, will exclude generated MD5s starting with "bbbbbbbbaaaaaaaa" + * @return list of executables in the repository + * @throws SQLException when issuing the SQL query + * @throws LSHException when creating the executable record + */ + private List queryExecutables(DescriptionManager manager, int limit, + String filterMd5, String filterExeName, long filterArch, long filterCompilerName, + ExeTableOrderColumn columnName, boolean includeFakes) + throws SQLException, LSHException { + + List records = new ArrayList<>(); + + List rows = exeTable.queryAllExe(limit, filterMd5, filterExeName, filterArch, + filterCompilerName, columnName, includeFakes); + + for (ExecutableRow row : rows) { + ExecutableRecord record = exeTable.makeExecutableRecord(manager, row); + records.add(record); + } + return records; + } + + private ExecutableRecord findSingleExecutable(ExeSpecifier spec, DescriptionManager manager) + throws SQLException, LSHException { + if (!StringUtils.isBlank(spec.exemd5)) { + ExecutableRow row = exeTable.queryMd5ExeMatch(spec.exemd5); + if (row == null) { + return null; + } + return exeTable.makeExecutableRecord(manager, row); + } + if (StringUtils.isBlank(spec.exename)) { + throw new LSHException("ExeSpecifier must provide either md5 or name"); + } + return exeTable.querySingleExecutable(manager, spec.exename, spec.arch, spec.execompname); + } + + private ExecutableRecord findSingleExeWithMap(ExeSpecifier exe, DescriptionManager manager, + TreeMap nameMap) throws SQLException, LSHException { + ExecutableRecord erec = nameMap.get(exe); + if (erec != null) { + return erec; + } + erec = findSingleExecutable(exe, manager); + nameMap.put(exe, erec); // Cache ExecutableRecord in map, even if its null + return erec; + } + + /** + * Query for an executable via its md5 hash + * @param md5 is the 32-character md5 hash + * @param res is DescriptionManager container to hold the result + * @return the ExecutableRecord is it was found, null otherwise + * @throws LSHException if executable record creation fails + * @throws SQLException if there is an error issuing the query + */ + private ExecutableRecord queryExecutableByMd5(String md5, DescriptionManager res) + throws LSHException, SQLException { + ExecutableRow row = exeTable.queryMd5ExeMatch(md5); + if (row != null) { + return exeTable.makeExecutableRecord(res, row); + } + return null; + } + + /** + * For every function currently in the manager, fill in its call graph information + * @param manage the executable descriptor + * @throws LSHException if the database does not track call graph information + * @throws SQLException if there is a problem querying for function information + */ + private void queryCallgraph(DescriptionManager manage) throws LSHException, SQLException { + if (!info.trackcallgraph) { + throw new LSHException("Database does not track callgraph"); + } + TreeMap funcmap = new TreeMap<>(); + manage.generateFunctionIdMap(funcmap); + List funclist = new ArrayList<>(); + for (FunctionDescription element : funcmap.values()) { // Build a static copy of the list of functions + funclist.add(element); + } + for (FunctionDescription element : funclist) { + fillinChildren(element, manage, funcmap); + } + } + + protected QueryResponseRecord doQuery(BSimQuery query, Connection c) + throws LSHException, SQLException, DatabaseNonFatalException { + if (query instanceof QueryNearest q) { + fdbQueryNearest(q); + } + else if (query instanceof QueryNearestVector q) { + fdbQueryNearestVector(q); + } + else if (query instanceof InsertRequest q) { + fdbDatabaseInsert(q); + } + else if (query instanceof QueryInfo q) { + fdbDatabaseInfo(q); + } + else if (query instanceof QueryName q) { + fdbQueryName(q); + } + else if (query instanceof QueryExeInfo q) { + fdbQueryExeInfo(q); + } + else if (query instanceof QueryExeCount q) { + fdbQueryExeCount(q); + } + else if (query instanceof CreateDatabase q) { + fdbDatabaseCreate(q); + } + else if (query instanceof QueryChildren q) { + fdbQueryChildren(q); + } + else if (query instanceof QueryDelete q) { + fdbDelete(q); + } + else if (query instanceof QueryUpdate q) { + fdbUpdate(q); + } + else if (query instanceof QueryVectorId q) { + fdbQueryVectorId(q); + } + else if (query instanceof QueryVectorMatch q) { + fdbQueryVectorMatch(q); + } + else if (query instanceof QueryPair q) { + fdbQueryPair(q); + } + else if (query instanceof QueryOptionalValues q) { + fdbQueryOptionalValues(q); + } + else if (query instanceof InsertOptionalValues q) { + fdbInsertOptionalValues(q); + } + else if (query instanceof QueryOptionalExist q) { + fdbOptionalExist(q); + } + else if (query instanceof InstallCategoryRequest q) { + fdbInstallCategory(q); + } + else if (query instanceof InstallTagRequest q) { + fdbInstallTag(q); + } + else if (query instanceof InstallMetadataRequest q) { + fdbInstallMetadata(q); + } + else { + return null; + } + return query.getResponse(); + } + + @Override + public QueryResponseRecord query(BSimQuery query) { + + lasterror = null; + try { + if (!(query instanceof CreateDatabase) && !initialize()) { + lasterror = new Error(ErrorCategory.Nodatabase, "The database does not exist"); + return null; + } + + query.buildResponseTemplate(); + QueryResponseRecord response = doQuery(query, db); + if (response == null) { + lasterror = new Error(ErrorCategory.Fatal, "Unknown query type"); + query.clearResponse(); + } + } + catch (DatabaseNonFatalException err) { + lasterror = new Error(ErrorCategory.Nonfatal, + "Skipping -" + query.getName() + "- : " + err.getMessage()); + query.clearResponse(); + } + catch (LSHException err) { + lasterror = new Error(ErrorCategory.Fatal, + "Fatal error during -" + query.getName() + "- : " + err.getMessage()); + query.clearResponse(); + } + catch (SQLException err) { + lasterror = new Error(ErrorCategory.Fatal, + "SQL error during -" + query.getName() + "- : " + err.getMessage()); + query.clearResponse(); + } + return query.getResponse(); + } + + /** + * Entry point for the QueryNearestVector command + * + * @param query the query to execute + * @throws LSHException if the provided signature data is invalid + * @throws SQLException if there is an error issuing the query + */ + private void fdbQueryNearestVector(QueryNearestVector query) throws LSHException, SQLException { + FunctionDatabase.checkSettingsForQuery(query.manage, info); + queryNearestVector(query); + } + + /** + * Entry point for the QueryPair command + * + * @param query the query to execute + * @throws SQLException if there is an error issuing the query + * @throws LSHException if no md5 or exe name is provided + */ + private void fdbQueryPair(QueryPair query) throws SQLException, LSHException { + ResponsePair response = query.pairResponse; + ResponsePair.Accumulator accumulator = new ResponsePair.Accumulator(); + + List aFuncList = new ArrayList<>(); + List bFuncList = new ArrayList<>(); + DescriptionManager resManage = new DescriptionManager(); + TreeMap nameMap = new TreeMap<>(); + for (PairInput pairInput : query.pairs) { + FunctionDescription funcA = null; + FunctionDescription funcB = null; + ExecutableRecord erec = findSingleExeWithMap(pairInput.execA, resManage, nameMap); + if (erec == null) { + accumulator.missedExe += 1; + } + else { + funcA = queryByNameAddress(resManage, erec, pairInput.funcA.funcName, + pairInput.funcA.address, true); + if (funcA == null) { + accumulator.missedFunc += 1; + } + } + + erec = findSingleExeWithMap(pairInput.execB, resManage, nameMap); + if (erec == null) { + accumulator.missedExe += 1; + } + else { + funcB = queryByNameAddress(resManage, erec, pairInput.funcB.funcName, + pairInput.funcB.address, true); + if (funcB == null) { + accumulator.missedFunc += 1; + } + } + aFuncList.add(funcA); + bFuncList.add(funcB); + } + + Iterator bIter = bFuncList.iterator(); + VectorCompare vectorData = new VectorCompare(); + for (FunctionDescription funcA : aFuncList) { + FunctionDescription funcB = bIter.next(); + if (funcA == null || funcB == null) { + continue; + } + SignatureRecord sigA = funcA.getSignatureRecord(); + if (sigA == null) { + accumulator.missedVector += 1; + continue; + } + SignatureRecord sigB = funcB.getSignatureRecord(); + if (sigB == null) { + accumulator.missedVector += 1; + continue; + } + double sim = sigA.getLSHVector().compare(sigB.getLSHVector(), vectorData); + double signif = vectorFactory.calculateSignificance(vectorData); + PairNote pairNote = new PairNote(funcA, funcB, sim, signif, vectorData.dotproduct, + vectorData.acount, vectorData.bcount, vectorData.intersectcount); + response.notes.add(pairNote); + accumulator.pairCount += 1; + accumulator.sumSim += sim; + accumulator.sumSimSquare += sim * sim; + accumulator.sumSig += signif; + accumulator.sumSigSquare += signif * signif; + } + response.scale = vectorFactory.getSignificanceScale(); + response.fillOutStatistics(accumulator); + } + + /** + * Entry point for the QueryNearest command + * + * @param query the query to execute + * @throws LSHException if there is a problem creating executable records + * @throws SQLException if there is an error issuing the query + */ + private void fdbQueryNearest(QueryNearest query) throws LSHException, SQLException { + FunctionDatabase.checkSettingsForQuery(query.manage, info); + BSimSqlClause filter = null; + if (query.bsimFilter != null) { + ExecutableRecord repexe = query.manage.getExecutableRecordSet().first(); + IDSQLResolution idres[] = new IDSQLResolution[query.bsimFilter.numAtoms()]; + for (int i = 0; i < idres.length; ++i) { + FilterAtom atom = query.bsimFilter.getAtom(i); + idres[i] = atom.type.generateIDSQLResolution(atom); + if (idres[i] != null) { + idres[i].resolve(this, repexe); + } + } + filter = SQLEffects.createFilter(query.bsimFilter, idres, this); + } + ResponseNearest response = query.nearresponse; + response.totalfunc = 0; + response.totalmatch = 0; + response.uniquematch = 0; + + DescriptionManager descMgr = new DescriptionManager(); + Iterator iter = query.manage.listAllFunctions(); + + queryFunctions(query, filter, response, descMgr, iter); + response.manage.transferSettings(query.manage); // Echo back the + // settings + if (query.fillinCategories && (info.execats != null)) { + fillinExecutableCategories(response.manage); + } + } + + /** + * Entry point for the QueryVectorId command + * + * @param query contains list of ids to query + * @throws SQLException if there is an error issuing the query + */ + private void fdbQueryVectorId(QueryVectorId query) throws SQLException { + for (Long id : query.vectorIds) { + VectorResult res = queryVectorId(id); + query.vectorIdResponse.vectorResults.add(res); + } + } + + /** + * Entry point for the QueryVectorMatch command + * + * @param query contains list of ids to match + * @throws SQLException if there is an error issuing the query + * @throws LSHException if there is a problem making executable records + */ + private void fdbQueryVectorMatch(QueryVectorMatch query) throws SQLException, LSHException { + BSimSqlClause filter = null; + if (query.bsimFilter != null) { + ExecutableRecord repexe = null; // Needed for the ExternalFunction filter + IDSQLResolution idres[] = new IDSQLResolution[query.bsimFilter.numAtoms()]; + for (int i = 0; i < idres.length; ++i) { + FilterAtom atom = query.bsimFilter.getAtom(i); + idres[i] = atom.type.generateIDSQLResolution(atom); + if (idres[i] != null) { + idres[i].resolve(this, repexe); + } + } + filter = SQLEffects.createFilter(query.bsimFilter, idres, this); + } + List vectorList = new ArrayList<>(); + for (Long id : query.vectorIds) { + VectorResult vecResult = queryVectorId(id); + vectorList.add(vecResult); + } + int count = 0; + DescriptionManager manage = query.matchresponse.manage; + for (VectorResult vecResult : vectorList) { + if (count >= query.max) { + break; + } + SignatureRecord srec = manage.newSignature(vecResult.vec, vecResult.hitcount); + List descres; + if (filter == null) { + descres = descTable.queryVectorIdMatch(vecResult.vectorid, query.max - count); + } + else { + descres = descTable.queryVectorIdMatchFilter(vecResult.vectorid, + filter.tableClause(), filter.whereClause(), query.max - count); + } + if (descres == null) { + throw new SQLException( + "Error querying vectorid: " + Long.toString(vecResult.vectorid)); + } + if (descres.size() == 0) { + if (filter != null) { + continue; // Filter may have eliminated all results + } + // Otherwise this is a sign of corruption in the database + throw new SQLException( + "No functions matching vectorid: " + Long.toString(vecResult.vectorid)); + } + count += descres.size(); + convertDescriptionRows(null, descres, vecResult, manage, srec); + } + if (query.fillinCategories && (info.execats != null)) { + fillinExecutableCategories(manage); + } + } + + /** + * @param query the query to execute + * @param filter the function filter + * @param response the response object + * @param descMgr the executable descriptor + * @param iter the function iterator + * @return the number of unique results found + * @throws SQLException if there is an error issuing the query + * @throws LSHException if there is a problem creating executable records + */ + protected int queryFunctions(QueryNearest query, BSimSqlClause filter, ResponseNearest response, + DescriptionManager descMgr, Iterator iter) + throws SQLException, LSHException { + + // Keep a map of the feature vectors and their query results; if we have vectors that + // are of equal value, we'll just query the first one, and use those results for the + // others. + HashMap> vecToResultMap = new HashMap<>(); + + while (iter.hasNext()) { + FunctionDescription frec = iter.next(); + SignatureRecord srec = frec.getSignatureRecord(); + + if (srec == null) { + continue; + } + LSHVector thevec = srec.getLSHVector(); + double len2 = vectorFactory.getSelfSignificance(thevec); + + // Self significance should be bigger than the significance threshold + // (or its impossible our result can exceed the threshold) + if (len2 < query.signifthresh) { + continue; + } + response.totalfunc += 1; + SimilarityResult simres = new SimilarityResult(frec); + if (descMgr.getExecutableRecordSet().size() > 1000) { + descMgr.clear(); + } + else { + // Try to preserve ExecutableRecords so we don't have to do SQL + // every time + descMgr.clearFunctions(); + } + queryNearest(simres, descMgr, thevec, query, filter, vecToResultMap); + if (simres.size() == 0) { + continue; + } + response.totalmatch += 1; + if (simres.size() == 1) { + response.uniquematch += 1; + } + + response.result.add(simres); + + simres.transfer(response.manage, true); + } + + return vecToResultMap.size(); + } + + /** + * Entry point for the InsertRequest command + * @param query the query to execute + * @throws LSHException if trying to insert into a read-only database + * @throws SQLException if there is an error issuing the query + * @throws DatabaseNonFatalException if there are duplicate executables and/or functions + */ + private void fdbDatabaseInsert(InsertRequest query) + throws LSHException, SQLException, DatabaseNonFatalException { + if (info.readonly) { + throw new LSHException("Trying to insert on read-only database"); + } + if (FunctionDatabase.checkSettingsForInsert(query.manage, info)) { // Check if settings are valid and is this is first insert + info.major = query.manage.getMajorVersion(); + info.minor = query.manage.getMinorVersion(); + info.settings = query.manage.getSettings(); + keyValueTable.writeBasicInfo(info); // Save off the settings associated with this first insert + } + ResponseInsert response = query.insertresponse; + if ((query.repo_override != null) && (query.repo_override.length() != 0)) { + query.manage.overrideRepository(query.repo_override, query.path_override); + } + insert(query.manage); + response.numexe = query.manage.getExecutableRecordSet().size(); + response.numfunc = query.manage.numFunctions(); + } + + /** + * Entry point for the QueryInfo command + * @param query the query to execute + */ + private void fdbDatabaseInfo(QueryInfo query) { + ResponseInfo response = query.inforesponse; + response.info = info; + } + + /** + * Entry point for the QueryName command + * @param query the query to execute + * @throws SQLException if there is an error issuing the query + * @throws LSHException if there is a problem query callgraph informatino + */ + private void fdbQueryName(QueryName query) throws SQLException, LSHException { + ResponseName response = query.nameresponse; + response.printselfsig = query.printselfsig; + response.printjustexe = query.printjustexe; + response.manage.setVersion(info.major, info.minor); + response.manage.setSettings(info.settings); + + ExecutableRecord erec = findSingleExecutable(query.spec, response.manage); + if (erec == null) { + response.uniqueexecutable = false; + return; + } + response.uniqueexecutable = true; + + queryByName(null, response.manage, erec, query.funcname, query.fillinSigs, query.maxfunc); + if (query.fillinCallgraph) { + queryCallgraph(response.manage); + } + if (query.fillinCategories && (info.execats != null)) { + fillinExecutableCategories(response.manage); + } + } + + /** + * Queries the database for all executables matching the search criteria in the given + * {@link QueryExeInfo} object. Results are stored in the query info object + * + * @param query the query information + * @throws SQLException if there is an error executing the query + * @throws LSHException if there is an error executing the query + */ + private void fdbQueryExeInfo(QueryExeInfo query) throws SQLException, LSHException { + ResponseExe response = query.exeresponse; + + boolean unknownArchOrCompiler = false; + + long archId = 0; + if (query.filterArch != null) { + archId = queryArchString(query.filterArch); + if (archId == 0) { + unknownArchOrCompiler = true; + Msg.warn(this, + "Architecture ID not defined within BSim database: " + query.filterArch); + } + } + + long compId = 0; + if (query.filterCompilerName != null) { + compId = queryCompilerString(query.filterCompilerName); + if (compId == 0) { + unknownArchOrCompiler = true; + Msg.warn(this, + "Compiler ID not defined within BSim database: " + query.filterCompilerName); + } + } + + if (unknownArchOrCompiler) { + response.records = List.of(); + response.recordCount = 0; + return; + } + + List records = + queryExecutables(response.manage, query.limit, query.filterMd5, query.filterExeName, + archId, compId, query.sortColumn, query.includeFakes); + response.records = records; + response.recordCount = records.size(); + if (query.fillinCategories && (info.execats != null)) { + fillinExecutableCategories(response.manage); + } + } + + /** + * Queries the database for the number of executables matching the filter criteria + * + * @param query the query information + * @throws SQLException if there is an error executing the query + */ + private void fdbQueryExeCount(QueryExeCount query) throws SQLException { + ResponseExe response = query.exeresponse; + long archId = 0; + if (query.filterArch != null) { + archId = queryArchString(query.filterArch); + } + long compId = 0; + if (query.filterCompilerName != null) { + compId = queryCompilerString(query.filterCompilerName); + } + response.recordCount = exeTable.queryExeCount(query.filterMd5, query.filterExeName, archId, + compId, query.includeFakes); + } + + /** + * Entry point for the QueryChildren command + * @param query the query to execute + * @throws LSHException if the database does not track callgraph or the function is not found + * @throws SQLException if there is an error issuing the query + */ + private void fdbQueryChildren(QueryChildren query) throws LSHException, SQLException { + if (!info.trackcallgraph) { + throw new LSHException("Database does not track callgraph"); + } + ResponseChildren response = query.childrenresponse; + ExecutableRecord exe = null; + + if (query.md5sum.length() != 0) { + exe = queryExecutableByMd5(query.md5sum, response.manage); + } + else { + exe = exeTable.querySingleExecutable(response.manage, query.name_exec, query.arch, + query.name_compiler); + if (exe == null) { + throw new LSHException("Could not (uniquely) match executable"); + } + } + for (FunctionEntry entry : query.functionKeys) { + FunctionDescription func = + queryByNameAddress(response.manage, exe, entry.funcName, entry.address, true); + if (func == null) { + throw new LSHException("Could not find function: " + entry.funcName); + } + response.correspond.add(func); + } + + TreeMap funcmap = new TreeMap<>(); + response.manage.generateFunctionIdMap(funcmap); + for (FunctionDescription element : response.correspond) { + fillinChildren(element, response.manage, funcmap); + } + } + + /** + * Entry point for the CreateDatabase command + * @param query the query to execute + * @throws LSHException if the configuration template cannot be loaded + * @throws SQLException if the {@link #generate(Configuration)} call fails + */ + private void fdbDatabaseCreate(CreateDatabase query) throws LSHException, SQLException { + ResponseInfo response = query.inforesponse; + Configuration config = FunctionDatabase.loadConfigurationTemplate(query.config_template); + // Copy in any overriding fields in the query + if (query.info.databasename != null) { + config.info.databasename = query.info.databasename; + } + if (query.info.owner != null) { + config.info.owner = query.info.owner; + } + if (query.info.description != null) { + config.info.description = query.info.description; + } + if (!query.info.trackcallgraph) { + config.info.trackcallgraph = query.info.trackcallgraph; + } + if (query.info.functionTags != null) { + checkStrings(query.info.functionTags, "function tags", + FunctionTagBSimFilterType.MAX_TAG_COUNT); + config.info.functionTags = query.info.functionTags; + } + if (query.info.execats != null) { + checkStrings(query.info.execats, "categories", -1); + config.info.execats = query.info.execats; + } + generate(config); + response.info = config.info; + } + + private static void checkStrings(List list, String type, int limit) + throws LSHException { + if (limit > 0 && list.size() > limit) { + throw new LSHException("Too many " + type + " specified (limit=" + + FunctionTagBSimFilterType.MAX_TAG_COUNT + "): " + list.size()); + } + Set names = new HashSet<>(); + for (String name : list) { + if (!CategoryRecord.enforceTypeCharacters(name)) { + throw new LSHException("Bad characters in one or more proposed " + type); + } + if (!names.add(name)) { + throw new LSHException("Duplicate " + type + " entry specified: " + name); + } + } + } + + /** + * Entry point for the InstallCategoryRequest command + * @param query the query to execute + * @throws LSHException if the category is invalid or already exists + * @throws SQLException if there is an error issuing the query + */ + private void fdbInstallCategory(InstallCategoryRequest query) + throws LSHException, SQLException { + ResponseInfo response = query.installresponse; + if (!CategoryRecord.enforceTypeCharacters(query.type_name)) { + throw new LSHException("Bad characters in proposed category type"); + } + if (query.isdatecolumn) { + info.dateColumnName = query.type_name; + keyValueTable.insert("datecolumn", info.dateColumnName); + response.info = info; + return; + } + // Check for existing category + if (info.execats != null) { + for (String cat : info.execats) { + if (cat.equals(query.type_name)) { + throw new LSHException("Executable category already exists"); + } + } + } + if (info.execats == null) { + info.execats = new ArrayList<>(); + } + info.execats.add(query.type_name); + keyValueTable.writeExecutableCategories(info); + response.info = info; + } + + /** + * Returns a list of all function tags in the database. + * + * @return list of function tags + */ + public List getFunctionTags() { + return info.functionTags; + } + + /** + * Entry point for the InstallTagRequest command + * @param query the install query to execute + * @throws LSHException if the function tag already exists or is not valid + * @throws SQLException if there is an error issuing the query + */ + private void fdbInstallTag(InstallTagRequest query) throws LSHException, SQLException { + ResponseInfo response = query.installresponse; + if (!CategoryRecord.enforceTypeCharacters(query.tag_name)) { + throw new LSHException("Bad characters in proposed function tag"); + } + // Check for existing tag + if (info.functionTags != null) { + if (info.functionTags.contains(query.tag_name)) { + throw new LSHException("Function tag already exists"); + } + } + if (info.functionTags == null) { + info.functionTags = new ArrayList<>(); + } + // There are only 32-bits of space in the function record reserved for storing the presence of tags + if (info.functionTags.size() >= FunctionTagBSimFilterType.MAX_TAG_COUNT) { + throw new LSHException( + "Cannot allocate new function tag: " + query.tag_name + " - Column space is full"); + } + info.functionTags.add(query.tag_name); + keyValueTable.writeFunctionTags(info); + response.info = info; + } + + /** + * Entry point for the InstallMetadataRequest command + * @param query the query to execute + * @throws SQLException if basic info can't be written to the table + */ + private void fdbInstallMetadata(InstallMetadataRequest query) throws SQLException { + ResponseInfo response = query.installresponse; + if (query.dbname != null) { + info.databasename = query.dbname; + } + if (query.owner != null) { + info.owner = query.owner; + } + if (query.description != null) { + info.description = query.description; + } + if (query.dbname != null || query.owner != null || query.description != null) { + keyValueTable.writeBasicInfo(info); + } + response.info = info; + } + + /** + * Entry point for the QueryDelete command + * @param query the query to execute + * @throws SQLException if there is an error issuing the query + * @throws LSHException if there is an error issuing the query + */ + private void fdbDelete(QueryDelete query) throws SQLException, LSHException { + ResponseDelete response = query.respdelete; + for (ExeSpecifier spec : query.exelist) { + DescriptionManager manage = new DescriptionManager(); + ExecutableRecord erec = null; + boolean commit = false; + beginTransaction(true); + try { + if (spec.exemd5 != null && spec.exemd5.length() != 0) { + ExecutableRow row = exeTable.queryMd5ExeMatch(spec.exemd5); + if (row != null) { + erec = exeTable.makeExecutableRecord(manage, row); + } + } + else { + erec = exeTable.querySingleExecutable(manage, spec.exename, spec.arch, + spec.execompname); + } + if (erec == null) { + response.missedlist.add(spec); + continue; + } + ResponseDelete.DeleteResult delrec = new ResponseDelete.DeleteResult(); + delrec.md5 = erec.getMd5(); + delrec.name = erec.getNameExec(); + List funclist = new ArrayList<>(); + queryAllFunc(funclist, erec, manage, 0); + Set table = IdHistogram.buildVectorIdHistogram(funclist.iterator()); + deleteVectors(table.iterator()); + delrec.funccount = deleteExecutable(erec, funclist, (info.execats != null)); + response.reslist.add(delrec); + commit = true; + } + finally { + endTransaction(commit); + } + } + } + + /** + * Entry point for the QueryUpdate command + * @param query the query to execute + * @throws LSHException if there is an error issuing the query + * @throws SQLException if there is an error issuing the query + */ + private void fdbUpdate(QueryUpdate query) throws LSHException, SQLException { + ResponseUpdate response = query.updateresponse; + for (ExecutableRecord erec : query.manage.getExecutableRecordSet()) { + int res = updateExecutable(query.manage, erec, response.badfunc, info.execats != null); + if (res < 0) { + response.badexe.add(erec); + } + else { + if ((res & 1) != 0) { + response.exeupdate += 1; + } + response.funcupdate += res >> 1; + } + } + } + + /** + * Entry point for QueryOptionalExist command + * @param query is the parameters for the query + * @throws SQLException for problems with the connection + */ + private void fdbOptionalExist(QueryOptionalExist query) throws SQLException { + ResponseOptionalExist response = query.optionalresponse; + OptionalTable table = + getOptionalTable(query.tableName, query.keyType, query.valueType, true); + response.tableExists = (table != null); + response.wasCreated = false; + if (table == null && query.attemptCreation) { + table = getOptionalTable(query.tableName, query.keyType, query.valueType, false); + table.createTable(); + response.wasCreated = true; + } + else if (table != null && query.clearTable) { + table.clearTable(); + } + } + + /** + * Entry point for the QueryOptionalValues command + * @param query is the parameters for the query + * @throws SQLException for problems with the connection + */ + private void fdbQueryOptionalValues(QueryOptionalValues query) throws SQLException { + ResponseOptionalValues response = query.optionalresponse; + OptionalTable table = + getOptionalTable(query.tableName, query.keyType, query.valueType, true); + if (table == null) { + response.resultArray = null; + response.tableExists = false; + return; + } + response.tableExists = true; + response.resultArray = new Object[query.keys.length]; + for (int i = 0; i < response.resultArray.length; ++i) { + response.resultArray[i] = table.readValue(query.keys[i]); + } + } + + /** + * Entry point for the InsertOptionalValues command + * @param query is the parameters for the command + * @throws SQLException for problems with the connection + */ + private void fdbInsertOptionalValues(InsertOptionalValues query) throws SQLException { + ResponseOptionalExist response = query.optionalresponse; + OptionalTable table = + getOptionalTable(query.tableName, query.keyType, query.valueType, true); + response.wasCreated = false; + if (table == null) { + response.tableExists = false; + return; + } + response.tableExists = true; + beginTransaction(false); + boolean commit = false; + try { + table.lockForWrite(); + for (int i = 0; i < query.keys.length; ++i) { + table.writeValue(query.keys[i], query.values[i]); + } + commit = true; + } + finally { + endTransaction(commit); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/BSimSqlClause.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/BSimSqlClause.java new file mode 100644 index 0000000000..b6887d285c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/BSimSqlClause.java @@ -0,0 +1,25 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +/** + * The SQL clauses for all the filters that are to be used in a BSim query + * @param tableClause the table SQL clause + * @param whereClause the where SQL clause + */ +public record BSimSqlClause(String tableClause, String whereClause) { + // +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/CancelledSQLException.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/CancelledSQLException.java new file mode 100644 index 0000000000..efcb5bb73f --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/CancelledSQLException.java @@ -0,0 +1,32 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.sql.SQLException; + +/** + * {@link CancelledSQLException} indicates a SQL operation was intentionally cancelled. + */ +public class CancelledSQLException extends SQLException { + + /** + * Constructor + * @param reason reason SQL operation was cancelled. + */ + public CancelledSQLException(String reason) { + super(reason); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/Configuration.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/Configuration.java new file mode 100755 index 0000000000..3d318c35d2 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/Configuration.java @@ -0,0 +1,101 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.io.*; + +import org.xml.sax.ErrorHandler; +import org.xml.sax.SAXException; + +import generic.jar.ResourceFile; +import generic.lsh.vector.IDFLookup; +import generic.lsh.vector.WeightFactory; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +public class Configuration { + public DatabaseInformation info; + public int k; // Number of bits in a bin id + public int L; // Number of separate binnings + public WeightFactory weightfactory; + public IDFLookup idflookup; + + public void saveXml(Writer fwrite) throws IOException { + fwrite.write("\n"); + info.saveXml(fwrite); + StringBuffer buf = new StringBuffer(); + buf.append("").append(k).append("\n"); + buf.append("").append(L).append("\n"); + fwrite.write(buf.toString()); + weightfactory.saveXml(fwrite); + idflookup.saveXml(fwrite); + fwrite.write("\n"); + } + + public void restoreXml(XmlPullParser parser) { + parser.start("dbconfig"); + info = new DatabaseInformation(); + info.restoreXml(parser); + parser.start("k"); + k = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("L"); + L = SpecXmlUtils.decodeInt(parser.end().getText()); + weightfactory = new WeightFactory(); + weightfactory.restoreXml(parser); + idflookup = new IDFLookup(); + idflookup.restoreXml(parser); + parser.end(); + } + + public void loadTemplate(ResourceFile rootPath, String filename) + throws SAXException, IOException { + ResourceFile file = new ResourceFile(rootPath, filename + ".xml"); + if (!file.exists()) { + throw new FileNotFoundException("Unable to find configuration template"); + } + ErrorHandler handler = SpecXmlUtils.getXmlHandler(); + XmlPullParser parser = + new NonThreadedXmlPullParserImpl(file.getInputStream(), file.getName(), handler, false); + parser.start("dbconfig"); + info = new DatabaseInformation(); + info.restoreXml(parser); + parser.start("k"); + k = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("L"); + L = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("weightsfile"); + String weightsfile = parser.end().getText(); + parser.end(); + weightfactory = new WeightFactory(); + idflookup = new IDFLookup(); + + if (weightsfile.equals("default")) { + return; // Use the default weights + } + file = new ResourceFile(rootPath, weightsfile); + if (!file.exists()) { + throw new FileNotFoundException("Unable to find weights file: "+weightsfile); + } + parser = + new NonThreadedXmlPullParserImpl(file.getInputStream(), file.getName(), handler, false); + parser.start("weights"); + weightfactory.restoreXml(parser); + idflookup.restoreXml(parser); + parser.end(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableComparison.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableComparison.java new file mode 100755 index 0000000000..785cb263e6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableComparison.java @@ -0,0 +1,513 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.util.*; +import java.util.Map.Entry; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.client.tables.ExeTable.ExeTableOrderColumn; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.*; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +/** + * Compare an entire set of executables to each other by combining + * significance scores between functions. If individual functions + * demonstrate multiple similarities, its score contributions are not + * over counted, and the final scores are symmetric. Scoring is efficient + * because it iterates over the precomputed clusters of similar functions + * in a BSim database. The algorithm does divide and conquer based on + * clusters of similar functions, which greatly improves efficiency over + * full quadratic comparison of all functions. This can be further bounded + * by putting a threshold on how close functions have to be to be considered + * in the same cluster and on how many functions can be in a cluster before + * ignoring their score contributions. + */ +public class ExecutableComparison { + private FunctionDatabase database; // Connection to the database + private ExecutableScorer scorer; // The scoring matrix and mapping to executables + private String singleMd5; + private TreeSet baseIds; // Set of relevant vector ids + private TreeSet queriedIds; // Set of vector ids that have already been queried + private LSHVectorFactory vectorFactory; // Factory for computing significance scores + private int hitCountThreshold; // Threshold for functions within a single cluster + private int maxHitCount; // Maximum hitcount seen (after performScoring) + private int exceedCount; // Number of times hitCountThreshold was exceeded + private TaskMonitor monitor; // Monitor for long running jobs + + /** + * Mutable integer class for histogram + */ + static public class Count { + public int value = 0; + } + + /** + * Initialize a comparison object with an active connection and thresholds, using the matrix scorer + * @param database is the active connection to a BSim database + * @param hitCountThreshold is the maximum number of functions to consider in one cluster + * @param monitor is a monitor to provide progress and cancellation checks + * @throws LSHException if the database connection is not established + */ + public ExecutableComparison(FunctionDatabase database, int hitCountThreshold, + TaskMonitor monitor) throws LSHException { + this.database = database; + this.singleMd5 = null; + this.hitCountThreshold = hitCountThreshold; + this.monitor = monitor; + if (this.monitor == null) { + this.monitor = TaskMonitor.DUMMY; + } + baseIds = null; + queriedIds = null; + DatabaseInformation info = pullConnectionInfo(); + scorer = new ExecutableScorer(); // The matrix scorer, everybody compared to everybody + scorer.transferSettings(info); + } + + /** + * Initialize a comparison object with an active connection and thresholds, using the row scorer + * @param database is the active connection to a BSim database + * @param hitCountThreshold is the maximum number of functions to consider in one cluster + * @param md5 is the 32-character md5 string of the executable to single out for comparison + * @param cache holds the self-scores or is null if normalized scores aren't needed + * @param monitor is a monitor to provide progress and cancellation checks + * @throws LSHException if the database connection is not established + */ + public ExecutableComparison(FunctionDatabase database, int hitCountThreshold, String md5, + ScoreCaching cache, TaskMonitor monitor) throws LSHException { + this.database = database; + this.singleMd5 = md5; + this.hitCountThreshold = hitCountThreshold; + this.monitor = monitor; + if (this.monitor == null) { + this.monitor = TaskMonitor.DUMMY; + } + baseIds = null; + queriedIds = null; + DatabaseInformation info = pullConnectionInfo(); + scorer = new ExecutableScorerSingle(cache); // The row scorer, compare single exe to everybody else + scorer.transferSettings(info); + addExecutable(singleMd5); + } + + /** + * @return maximum hit count seen for a cluster + */ + public int getMaxHitCount() { + return maxHitCount; + } + + /** + * @return number of clusters that exceeded hitCountThreshold + */ + public int getExceedCount() { + return exceedCount; + } + + /** + * @return true if similarity and significance thresholds have been set + */ + public boolean isConfigured() { + return (scorer.simThreshold > 0.0); + } + + /** + * Make sure the database has an active connection. Query for basic + * information and inform the scorer of settings + * @return information from the database + * @throws LSHException if something is wrong with the connection + */ + private DatabaseInformation pullConnectionInfo() throws LSHException { + if (!database.initialize()) { + throw new LSHException("Unable to connect to server"); + } + QueryInfo query = new QueryInfo(); + ResponseInfo response = query.execute(database); + if (response == null) { + throw new LSHException(database.getLastError().message); + } + vectorFactory = database.getLSHVectorFactory(); + return response.info; + } + + /** + * Look up a full ExecutableRecord in the database given an md5 + * @param md5 is the md5 String + * @return the corresponding ExecutableRecord + * @throws LSHException if there are problems accessing the database + * or the executable doesn't exist + */ + private ExecutableRecord lookupExecutable(String md5) throws LSHException { + QueryName query = new QueryName(); + query.spec = new ExeSpecifier(); + query.spec.exemd5 = md5; + query.maxfunc = 1; + query.fillinCallgraph = false; + query.fillinCategories = false; + query.fillinSigs = false; + ResponseName response = query.execute(database); + if (response == null) { + throw new LSHException(database.getLastError().message); + } + if (response.manage.numExecutables() != 1) { + throw new LSHException("Could not find executable"); + } + return response.manage.getExecutableRecordSet().first(); + } + + /** + * Query for all the vector ids associated with a specific executable. + * Store them in the vectorMap + * @param exeSpec indicates the specific executable + * @param histogram is non-null, provide a histogram of the vector ids + */ + private void pullVectorsForExe(ExeSpecifier exeSpec, Map histogram) { + QueryName queryName = new QueryName(); + queryName.spec = exeSpec; + // TODO: Need to make more of an effort to collect all vectors for large executables. + // But, this requires a change to the QueryName API to allow a window to be specified + queryName.maxfunc = 100000; + queryName.fillinCallgraph = false; + queryName.fillinCategories = false; + queryName.fillinSigs = false; + + ResponseName response = queryName.execute(database); + Iterator iter = response.manage.listAllFunctions(); + if (histogram == null) { + while (iter.hasNext()) { + baseIds.add(iter.next().getVectorId()); + } + } + else { + while (iter.hasNext()) { + Long id = iter.next().getVectorId(); + Count count = histogram.computeIfAbsent(id, key -> new Count()); + count.value += 1; + } + } + } + + /** + * Given a set of executables established for scoring, load all + * of the associated vector ids into -vectorMap- + * @throws CancelledException if something trips the monitor + */ + private void pullVectorsForScoringSet() throws CancelledException { + baseIds = new TreeSet(); + queriedIds = new TreeSet(); + if (scorer instanceof ExecutableScorerSingle) { // If we are doing single exe version + ExeSpecifier specifier = new ExeSpecifier(); + specifier.exemd5 = singleMd5; + pullVectorsForExe(specifier, null); // only pull vectors for the one executable + return; + } + monitor.setMessage("Accumulating vector ids"); + TreeSet recordSet = scorer.executableSet.getExecutableRecordSet(); + monitor.initialize(recordSet.size()); + ExeSpecifier specifier = new ExeSpecifier(); + for (ExecutableRecord exeRecord : recordSet) { + specifier.exemd5 = exeRecord.getMd5(); + pullVectorsForExe(specifier, null); + monitor.checkCancelled(); + monitor.incrementProgress(1); + } + } + + /** + * Build a QueryNearestVector object for querying a single vector. + * This currently involves building a temporary "function" to hold the vector + * @param vector is the LSHVector to prepare the query for + * @param threshold defines how similar "close" vectors are + * @return the QueryNearestVector object + * @throws LSHException if (impossibly) there are problems creating the query object + */ + private QueryNearestVector buildVectorQuery(LSHVector vector, double threshold) + throws LSHException { + QueryNearestVector query = new QueryNearestVector(); + + ExecutableRecord exeRecord = query.manage.newExecutableRecord( + "bbbbaaaabbbbaaaabbbbaaaabbbbaaaa", null, null, null, null, null, null, null); + FunctionDescription function = + query.manage.newFunctionDescription("tmp", 0x1000L, exeRecord); + SignatureRecord signature = query.manage.newSignature(vector, 1); + query.manage.attachSignature(function, signature); + + query.manage.transferSettings(scorer.executableSet); + query.thresh = threshold; + return query; + } + + /** + * Look up a single vector by id + * @param id is the Long id of the vector + * @return the matching vector result + * @throws LSHException if the vector does not exist + */ + private VectorResult buildSeedVector(Long id) throws LSHException { + QueryVectorId query = new QueryVectorId(); + query.vectorIds.add(id); + ResponseVectorId response = query.execute(database); + if (response == null || response.vectorResults.size() != 1) { + throw new LSHException("Could not locate vector by id"); + } + return response.vectorResults.get(0); + } + + /** + * Pull one ID out of the workList, look-up its corresponding vector, and query for nearby vectors. + * Add IDs of the close vectors (that have not been seen before) to the workList + * Use vectorMap to keep track of what's been seen/queried before + * @param workList is the current list of IDs yet to be processed for the cluster + * @param threshold defines how similar "close" vectors are + * @return the VectorResult of the next ID + * @throws LSHException if something goes wrong during a query + */ + private VectorResult queryVectorForCluster(TreeMap workList, + double threshold) + throws LSHException { + Entry entry = workList.pollFirstEntry(); + VectorResult currentVector = entry.getValue(); + baseIds.remove(entry.getKey()); // Look-up vector and remove from to-do list + queriedIds.add(entry.getKey()); // Mark that this id has been queried + if (currentVector == null) { + currentVector = buildSeedVector(entry.getKey()); // If VectorResult isn't present, this must be a seed for the cluster + } + QueryNearestVector query = buildVectorQuery(currentVector.vec, threshold); + ResponseNearestVector response = query.execute(database); + if (response == null || response.result.size() != 1) { + throw new LSHException("Could not perform query on vector"); + } + Iterator iter = response.result.get(0).iterator(); + while (iter.hasNext()) { + VectorResult vecResult = iter.next(); + Long curId = vecResult.vectorid; + if (queriedIds.contains(curId)) { + continue; // Already queried + } + workList.put(curId, vecResult); + } + return currentVector; + } + + /** + * Starting with the first vector in -vectorMap- build the cluster of vectors that are within + * a given -threshold- of each other. The cluster is the "connected" component containing the + * first vector, where two vectors are "connected" if they are similar to each other within + * the threshold. + * @param cluster will hold the list of vectors in the cluster + * @param simThreshold is the similarity threshold defining "near" or "connected" + * @param sigThreshold is the minimum significance for contributing score + * @return the total number of functions associated with any vector in the cluster + * @throws LSHException is something goes wrong during database queries + */ + private int buildCluster(List cluster, double simThreshold, + double sigThreshold) + throws LSHException { + TreeMap workList = new TreeMap(); + Long firstKey = baseIds.pollFirst(); + workList.put(firstKey, null); + int hitCount = 0; + while (!workList.isEmpty()) { + VectorResult vectorInfo = queryVectorForCluster(workList, simThreshold); // Retrieve vector, add close vectors + if (sigThreshold < vectorFactory.getSelfSignificance(vectorInfo.vec)) { // If self-sig exceeds threshold + cluster.add(vectorInfo); // add it to the cluster + hitCount += vectorInfo.hitcount; + } + } + return hitCount; + } + + /** + * For each vector in a list -cluster-, query the database and populate a + * container (DescriptionManager) with the functions associated with the vector. + * Return the list of containers + * with that vector + * @param cluster is the list of vectors + * @return the list of DescriptionManager containers + * @throws LSHException if anything goes wrong with queries + */ + private List vectorToFunctions(List cluster) + throws LSHException { + List result = new ArrayList(cluster.size()); + for (int i = 0; i < cluster.size(); ++i) { + VectorResult vector = cluster.get(i); + QueryVectorMatch query = new QueryVectorMatch(); + query.fillinCategories = false; + query.max = vector.hitcount + 10; // vector.hitcount should be exact, but set threshold slightly + // higher so we can tell if we exceeded hitcount + query.vectorIds.add(vector.vectorid); + ResponseVectorMatch response = query.execute(database); + if (response == null) { + throw new LSHException(database.getLastError().message); + } +// if (response.manage.numFunctions() > vector.hitcount) { +// throw new LSHException("Function count exceeds vector hitcount -- possible database corruption"); +// } + scorer.labelAndFilter(response.manage); + result.add(response.manage); + } + return result; + } + + /** + * @return the ExecutableScorer to allow examination of scores + */ + public ExecutableScorer getScorer() { + return scorer; + } + + /** + * Register an executable to be scored + * @param md5 is the MD5 string of the executable + * @throws LSHException if the executable is not in the database + */ + public void addExecutable(String md5) throws LSHException { + ExecutableRecord exeRecord = lookupExecutable(md5); + scorer.addExecutable(exeRecord); + } + + /** + * Add all executables currently in the database to this object for comparison. + * @param limit is the max number of executables to compare against (if greater than zero) + * @throws LSHException for problems retrieving ExecutableRecords from the database + */ + public void addAllExecutables(int limit) throws LSHException { + QueryExeInfo query = + new QueryExeInfo(limit, null, null, null, null, ExeTableOrderColumn.MD5, false); + ResponseExe responseExe = query.execute(database); + if (responseExe == null) { + throw new LSHException(database.getLastError().message); + } + for (ExecutableRecord exeRecord : responseExe.records) { + scorer.addExecutable(exeRecord); + } + } + + /** + * Perform scoring between all registered executables. + * @throws LSHException for any connection issues during the process + * @throws CancelledException if the monitor reports cancellation + */ + public void performScoring() throws LSHException, CancelledException { + maxHitCount = 0; + exceedCount = 0; + scorer.populateExecutableIndex(); + if (singleMd5 != null) { + scorer.setSingleExecutable(singleMd5); + } + pullVectorsForScoringSet(); + scorer.initializeScores(); + + monitor.setMessage("Processing similar functions"); + int maxSize = baseIds.size(); + monitor.initialize(maxSize); + if (scorer.simThreshold < 0.0) { + throw new LSHException("No thresholds have been established"); + } + while (!baseIds.isEmpty()) { + List vectors = new ArrayList(); + int hitcount = buildCluster(vectors, scorer.simThreshold, scorer.sigThreshold); + if (hitcount == 0) { // Zero is possible, if all vectors in cluster are below sig threshold + continue; + } + else if (hitcount > maxHitCount) { // Keep track of biggest hitcount + maxHitCount = hitcount; + } + if (!scorer.checkPreliminaryPairThreshold(hitcount, hitCountThreshold)) { // Cluster is too big + exceedCount += 1; // Count the occurrence + continue; // Don't score with this cluster + } + List vec2Functions = vectorToFunctions(vectors); + if (!scorer.scoreCluster(vectorFactory, vec2Functions, vectors, hitcount, + hitCountThreshold)) { + exceedCount += 1; + continue; + } + monitor.checkCancelled(); + monitor.setProgress(maxSize - baseIds.size()); + } + baseIds = null; + queriedIds = null; // Release storage + } + + /** + * Remove any old scores and set new thresholds for the scorer + * @param simThreshold is the similarity threshold for new scores + * @param sigThreshold is the significance threshold for new scores + * @throws LSHException if there are problems saving new thresholds + */ + public void resetThresholds(double simThreshold, double sigThreshold) throws LSHException { + scorer.resetStorage(simThreshold, sigThreshold); + } + + /** + * Generate any missing self-scores within the list of registered executables. + * @throws LSHException for problems retrieving vectors + * @throws CancelledException if the user clicks "cancel" + */ + public void fillinSelfScores() throws LSHException, CancelledException { + if (!(scorer instanceof ExecutableScorerSingle)) { + return; + } + ExecutableScorerSingle singleScorer = (ExecutableScorerSingle) scorer; + List missing = new ArrayList(); + singleScorer.prefetchSelfScores(missing); + int size = missing.size(); + if (size == 0) { + return; + } + if (size == 1) { + if (missing.get(0).getMd5().equals(singleMd5)) { + return; + } + } + + double sigThreshold = singleScorer.getSigThreshold(); + monitor.setMessage("Generating self-significance scores"); + monitor.initialize(size); + Iterator iter = missing.iterator(); + ExeSpecifier exeSpec = new ExeSpecifier(); + while (iter.hasNext()) { + ExecutableRecord exeRec = iter.next(); + if (exeRec.getMd5().equals(singleMd5)) { // Don't need to prefetch the singular executable + continue; + } + TreeMap histogram = new TreeMap(); + exeSpec.exemd5 = exeRec.getMd5(); + pullVectorsForExe(exeSpec, histogram); + double score = 0.0; + Iterator> histIter = histogram.entrySet().iterator(); + while (histIter.hasNext()) { + Entry entry = histIter.next(); + VectorResult vecResult = buildSeedVector(entry.getKey()); + double significance = vectorFactory.getSelfSignificance(vecResult.vec); + if (significance < sigThreshold) { + continue; + } + score += significance * entry.getValue().value; + } + scorer.commitSelfScore(exeRec.getMd5(), (float) score); + monitor.checkCancelled(); + monitor.incrementProgress(1); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableScorer.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableScorer.java new file mode 100755 index 0000000000..072068b476 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableScorer.java @@ -0,0 +1,521 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import generic.lsh.vector.VectorCompare; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; + +/** + * Class for accumulating a matrix of scores between pairs of executables + * ExecutableRecords are registered with addExecutable. Scoring is accumulated + * by repeatedly providing clusters of functions to scoreCluster. + */ +public class ExecutableScorer { + /** + * Container for a pair of FunctionDescriptions, possibly from different DescriptionManagers + * along with similarity/significance information + */ + public static class FunctionPair implements Comparable { + protected FunctionDescription funcA; + protected FunctionDescription funcB; + protected double similarity; + protected double significance; + + public FunctionPair(FunctionDescription a, FunctionDescription b, double sim, double sig) { + // For function pairs from the same two executables, make sure functions + // from the same executable are consistently on the same side + if (a.getExecutableRecord().getXrefIndex() <= b.getExecutableRecord().getXrefIndex()) { + funcA = a; + funcB = b; + } + else { + funcA = b; + funcB = a; + } + similarity = sim; + significance = sig; + } + + @Override + public int compareTo(FunctionPair o) { + int comp = Integer.compare(funcA.getExecutableRecord().getXrefIndex(), + o.funcA.getExecutableRecord().getXrefIndex()); + if (comp != 0) { // Compare first by side A executable + return comp; + } + comp = Integer.compare(funcB.getExecutableRecord().getXrefIndex(), + o.funcB.getExecutableRecord().getXrefIndex()); + if (comp != 0) { // Compare second by side B executable + return comp; + } + comp = Double.compare(similarity, o.similarity); + if (comp != 0) { + return -comp; // Compare third by similarity, bigger comes earlier + } + comp = Long.compare(funcA.getAddress(), o.funcA.getAddress()); + if (comp != 0) { // Compare fourth by address of side A function + return comp; + } + comp = Long.compare(funcB.getAddress(), o.funcB.getAddress()); + return comp; // Compare last by address of side B function + } + } + + protected DescriptionManager executableSet; // Set of executables to compare + protected Map index2ExeMap; // Mapping from index to registered executable + private float[][] score; // Matrix of accumulated scores + protected double simThreshold; // Similarity threshold associated with scores + protected double sigThreshold; // Significance threshold associated with scores + protected ExecutableRecord singleExe; // singled out executable + protected int singleExeXref; // index of the singled out executable to filter on + + public ExecutableScorer() { + executableSet = new DescriptionManager(); + score = null; + index2ExeMap = null; + singleExe = null; + singleExeXref = -1; + simThreshold = -1.0; + sigThreshold = -1.0; + } + + /** + * @return the similarity threshold associated with these scores + * OR -1.0 if no threshold has been set + */ + public double getSimThreshold() { + return simThreshold; + } + + /** + * @return the significance threshold associated with these scores + */ + public double getSigThreshold() { + return sigThreshold; + } + + /** + * Set a single executable as focus to enable the single parameter getScore(int) + * @param md5 is the 32-character md5 hash of the executable single out + * @throws LSHException if we can't find the executable + */ + public void setSingleExecutable(String md5) throws LSHException { + singleExe = executableSet.findExecutable(md5); + singleExeXref = singleExe.getXrefIndex(); // Save the xref of the single + } + + /** + * @return number of executable self-significance scores are (will be) available + */ + public int countSelfScores() { + return executableSet.numExecutables(); + } + + /** + * Clear any persistent storage for self-significance scores, and establish new thresholds + * @param simThresh is the new similarity threshold + * @param sigThresh is the new significance threshold + * @throws LSHException if there's a problem clearing storage + */ + public void resetStorage(double simThresh, double sigThresh) throws LSHException { + simThreshold = simThresh; + sigThreshold = sigThresh; + score = null; // Throw out any old scores + } + + /** + * @return the number of executables being compared + */ + public int numExecutables() { + return executableSet.numExecutables(); + } + + /** + * @return ExecutableRecord being singled out for comparison + */ + public ExecutableRecord getSingularExecutable() { + return singleExe; + } + + public float getSingularSelfScore() { + return score[singleExeXref - 1][singleExeXref - 1]; + } + + /** + * Retrieve a specific ExecutableRecord by md5 + * @param md5 is the MD5 string + * @return the matching ExecutableRecord + * @throws LSHException if the ExecutableRecord isn't present + */ + public ExecutableRecord getExecutable(String md5) throws LSHException { + return executableSet.findExecutable(md5); + } + + /** + * Get the index-th executable. NOTE: The first index is 1 + * @param index of the executable to retrieve + * @return the ExecutableRecord describing the executable + */ + public ExecutableRecord getExecutable(int index) { + if (index2ExeMap == null) { + index2ExeMap = executableSet.generateExecutableXrefMap(); + } + return index2ExeMap.get(index); + } + + /** + * Save off information about database settings to inform later queries + * @param info is the information object returned by the database + */ + protected void transferSettings(DatabaseInformation info) { + executableSet.setVersion(info.major, info.minor); + executableSet.setSettings(info.settings); + } + + /** + * Register an executable for the scoring matrix + * @param exeRecord is the ExecutableRecord to register + * @throws LSHException if the executable was already registered + * with different metadata + */ + protected void addExecutable(ExecutableRecord exeRecord) throws LSHException { + executableSet.transferExecutable(exeRecord); // Transfer (re)sets xrefIndex to 0 + } + + /** + * Assuming all executables have been registered, establish index values + * for all executables to facilitate accessing the scoring matrix + */ + protected void populateExecutableIndex() { + executableSet.populateExecutableXref(); + } + + /** + * For every executable in the container -manage-, if the executable + * matches up with a registered executable, set its xref index to match + * the registered executables xref index, otherwise set it to zero + * to indicate the executable should be filtered + * @param manage is the container of ExecutableRecords to label + */ + protected void labelAndFilter(DescriptionManager manage) { + manage.matchAndSetXrefs(executableSet); + } + + /** + * Initialize the scoring matrix with zero. The matrix size + * is the number of executables registered with addExecutable() + */ + protected void initializeScores() { + int size = executableSet.numExecutables(); + score = new float[size][]; + for (int i = 0; i < size; ++i) { + score[i] = new float[i + 1]; + float row[] = score[i]; + for (int j = 0; j <= i; ++j) { + row[j] = 0.0f; + } + } + } + + /** + * Given a pair of score contributing functions that have been + * fully filtered, add the score into the matrix + * @param pair is the pair of functions + */ + protected void scorePair(FunctionPair pair) { + int indexA = pair.funcA.getExecutableRecord().getXrefIndex(); + int indexB = pair.funcB.getExecutableRecord().getXrefIndex(); + + if (indexB > indexA) { + int tmp = indexA; + indexA = indexB; + indexB = tmp; + } + score[indexA - 1][indexB - 1] += pair.significance; + } + + /** + * Return the similarity score between two executables + * @param a is the index matching getXrefIndex() of the first executable + * @param b is the index matching getXrefIndex() of the second executable + * @return the similarity score + */ + public float getScore(int a, int b) { + if (b > a) { + int tmp = a; + a = b; + b = tmp; + } + return score[a - 1][b - 1]; + } + + /** + * Retrieve the similarity score of an executable with itself + * @param a is the index of the executable + * @return its self-similarity score + * @throws LSHException if the score is not accessible + */ + public float getSelfScore(int a) throws LSHException { + return score[a - 1][a - 1]; + } + + /** + * Commit the singled out executables self-significance score to permanent storage + * @throws LSHException if there's a problem writing, or the operation isn't supported + */ + public void commitSelfScore() throws LSHException { + throw new LSHException("Cannot commit self-score with the matrix scorer"); + } + + /** + * Commit a self-significance score for a specific executable to permanent storage + * @param md5 is the 32-character md5 hash of the executable + * @param selfScore is the self-significance score + * @throws LSHException if there's a problem writing, or the operation isn't supported + */ + protected void commitSelfScore(String md5, float selfScore) throws LSHException { + throw new LSHException("Cannot commit self-score with the matrix scorer"); + } + + /** + * Get score of executable (as compared to our singled out executable) + * @param a is the index of the executable + * @return the score + */ + public float getScore(int a) { + return getScore(singleExeXref, a); + } + + /** + * Computes a score comparing two executables, normalized between 0.0 and 1.0, + * indicating the percentage of functional similarity between the two. + * 1.0 means "identical" 0.0 means completely "dissimilar" + * @param a is the index of the first executable + * @param b is the index of the second executable + * @param useLibrary is true if the score measures percent "containment" + * of the smaller executable in the larger. + * @return the normalized score + * @throws LSHException if the self-scores for either executable are not available + */ + public float getNormalizedScore(int a, int b, boolean useLibrary) throws LSHException { + float baseScore = getScore(a, b); + float selfA = getSelfScore(a); + float selfB = getSelfScore(b); + // If selfA is bigger use selfB for library, selfA otherwise + if (selfA < selfB) { // If selfB is bigger + useLibrary = !useLibrary; // flip this + } + if (useLibrary) { + if (selfB == 0.0f) { + return -1.0f; + } + return baseScore / selfB; + } + if (selfA == 0.0) { + return -1.0f; + } + return baseScore / selfA; + } + + public float getNormalizedScore(int a, boolean useLibrary) throws LSHException { + return getNormalizedScore(a, singleExeXref, useLibrary); + } + + /** + * Generate all pairs of functions for any function associated with a list of vectors + * For each pair of functions generate the FunctionPair object with corresponding + * similarity and significance. This is inherently quadratic, but we try to be efficient. + * Duplicate vector pairs are only compared once and the similarity cached, + * and each function pair is generated only once, ie. (funcA,funcB) but not (funcB,funcA) + * @param vectorFactory provides weights for significance scores + * @param vec2func is the list of FunctionDescription sets associated with each vector + * @param vectors is the list of vectors + * @param hitcount is the cumulative total of functions + * @param pairThreshold is the maximum number of pairs that can be produced + * @return the array of FunctionPairs or null if pairThreshold is exceeded + */ + protected List pairFunctions(LSHVectorFactory vectorFactory, + List vec2func, List vectors, int hitcount, + int pairThreshold) { + int totalSize = hitcount * (hitcount + 1) / 2; + if (totalSize > pairThreshold) { + return null; + } + List result = new ArrayList(totalSize); + VectorCompare vectorCompare = new VectorCompare(); + for (int v1 = 0; v1 < vec2func.size(); ++v1) { + for (int v2 = v1; v2 < vec2func.size(); ++v2) { + double similarity = vectors.get(v1).vec.compare(vectors.get(v2).vec, vectorCompare); + if (similarity < simThreshold) { + continue; + } + double significance = vectorFactory.calculateSignificance(vectorCompare); + if (significance < sigThreshold) { + continue; + } + if (v1 == v2) { + Iterator iter1 = vec2func.get(v1).listAllFunctions(); + while (iter1.hasNext()) { + FunctionDescription func1 = iter1.next(); + int func1Index = func1.getExecutableRecord().getXrefIndex(); + if (func1Index == 0) { + continue; // Executable is not in our scoring set + } + FunctionDescription func2; + Iterator iter2 = vec2func.get(v1).listAllFunctions(); + do { + func2 = iter2.next(); + int func2Index = func2.getExecutableRecord().getXrefIndex(); + if (func2Index == 0) { + continue; // Executable is not in our scoring set + } + result.add(new FunctionPair(func1, func2, similarity, significance)); + } + while (func2 != func1); + } + } + else { + Iterator iter1 = vec2func.get(v1).listAllFunctions(); + while (iter1.hasNext()) { + FunctionDescription func1 = iter1.next(); + int func1Index = func1.getExecutableRecord().getXrefIndex(); + if (func1Index == 0) { + continue; // Executable is not in our scoring set + } + Iterator iter2 = vec2func.get(v2).listAllFunctions(); + while (iter2.hasNext()) { + FunctionDescription func2 = iter2.next(); + int func2Index = func2.getExecutableRecord().getXrefIndex(); + if (func2Index == 0) { + continue; // Executable is not in our scoring set + } + result.add(new FunctionPair(func1, func2, similarity, significance)); + } + } + } + } + } + return result; + } + + /** + * For function pairs between the same two executables, do the final filtering + * to create symmetric score contributions and accumulate the contributions in the matrix + * @param pairs is the full list of pairs in the cluster + * @param i is the first function pair sharing this pair of executables + * @param j is the the last(+1) function pair sharing this pair of executables + */ + private void scoreAcrossExecutablePair(List pairs, int i, int j) { + int size = j - i; + FunctionPair pair1 = pairs.get(i); + if (size == 1) { // If only one pair between the two executable + scorePair(pair1); // we can score it immediately + } + else if (size == 2) { // If there are two pairs between the two executables + FunctionPair pair2 = pairs.get(i + 1); + if (pair1.funcA == pair2.funcA || pair1.funcB == pair2.funcB) { // If there is a function in common + scorePair(pair1); // only score one of the pairs + } + else { + scorePair(pair1); // otherwise we can score both pairs + scorePair(pair2); + } + } + else if (pair1.funcA.getExecutableRecord() + .getXrefIndex() == pair1.funcB.getExecutableRecord().getXrefIndex()) { + // Pairs coming from the same executable + for (; i < j; ++i) { + FunctionPair pair = pairs.get(i); + if (pair.funcA == pair.funcB) { // Only score function paired with itself + scorePair(pair); + } + } + } + else { + HashSet aUsed = new HashSet(); // Each function can be score only once (for these executables) + HashSet bUsed = new HashSet(); // Keep track of which functions are used + for (; i < j; ++i) { // Run through the pairs (highest similarity first) + FunctionPair pair = pairs.get(i); + Long aAddress = pair.funcA.getAddress(); + if (aUsed.contains(aAddress)) { // If the left-side function has been used before + continue; // skip this pair + } + Long bAddress = pair.funcB.getAddress(); + if (bUsed.contains(bAddress)) { // If the right-side function has been used before + continue; // skip this pair + } + aUsed.add(aAddress); // Mark the two functions as used + bUsed.add(bAddress); + scorePair(pair); + } + } + } + + /** + * Make check if we are going to have too many pairs. + * This is preliminary because we haven't yet fetched the functions + * @param hitcount is the total number of pairs to fetch + * @param pairThreshold is the maximum number of pairs allowed + * @return true if the pair threshold is not exceeded + */ + protected boolean checkPreliminaryPairThreshold(int hitcount, int pairThreshold) { + int totalSize = hitcount * (hitcount + 1) / 2; + return (totalSize <= pairThreshold); + } + + /** + * Given a cluster of vectors, the set of functions associated with each vector, + * a similarity threshold, and total number of functions in the cluster, + * let each pair of function contribute to the score matrix + * @param vectorFactory is a factory for computing significance scores + * @param vec2Functions is the list of sets of functions + * @param vectors is the list of vectors + * @param hitcount is the number of functions in the cluster + * @param pairThreshold is maximum number of pairs allowed + * @return true if the number of pairs was not exceeded + */ + protected boolean scoreCluster(LSHVectorFactory vectorFactory, + List vec2Functions, List vectors, + int hitcount, int pairThreshold) { + List pairs = + pairFunctions(vectorFactory, vec2Functions, vectors, hitcount, pairThreshold); + if (pairs == null) { + return false; + } + Collections.sort(pairs); + int i = 0; + while (i < pairs.size()) { + ExecutableRecord rec1 = pairs.get(i).funcA.getExecutableRecord(); + ExecutableRecord rec2 = pairs.get(i).funcB.getExecutableRecord(); + int j = i + 1; + while (j < pairs.size()) { + FunctionPair currentPair = pairs.get(j); + if (!rec2.equals(currentPair.funcB.getExecutableRecord()) || + !rec1.equals(currentPair.funcA.getExecutableRecord())) { + break; + } + j += 1; + } + scoreAcrossExecutablePair(pairs, i, j); + i = j; + } + return true; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableScorerSingle.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableScorerSingle.java new file mode 100755 index 0000000000..3991f81dfa --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ExecutableScorerSingle.java @@ -0,0 +1,250 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.util.*; + +import generic.lsh.vector.*; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; + +/** + * ExecutableComparison scorer to use when we are comparing exactly one executable + * against a set of other executables (one to many). We override the {@link ExecutableScorer} + * (compare many to many) so that it effectively accesses only a single row + * of the scoring matrix to get the "one to many" behavior we want. + * The getNormalizedScore() methods on the base class require that executable self-scores, + * other than the singled-out executable's self-score, be cached in some way. + * Thus this scorer needs a {@link ScoreCaching} class. + */ +public class ExecutableScorerSingle extends ExecutableScorer { + + private float singleScore[]; // Score of single against all other executables + private ScoreCaching scoreCache; + + /** + * Construct the scorer. If normalized scores are required, a self-score cacher + * must be provided. + * @param cache is the self-score cacher or null + * @throws LSHException for problems initializing the cache + */ + public ExecutableScorerSingle(ScoreCaching cache) throws LSHException { + super(); // Turn on single executable filtering + singleScore = null; + scoreCache = cache; + if (scoreCache == null) { + scoreCache = new TemporaryScoreCaching(); + } + simThreshold = scoreCache.getSimThreshold(); + sigThreshold = scoreCache.getSigThreshold(); + } + + @Override + public void setSingleExecutable(String md5) throws LSHException { + if (singleExeXref >= 0) { + throw new LSHException("Cannot reset singled executable"); + } + super.setSingleExecutable(md5); + } + + @Override + public int countSelfScores() { + Iterator iter = executableSet.getExecutableRecordSet().iterator(); + int count = 0; + while (iter.hasNext()) { + ExecutableRecord exe = iter.next(); + try { + scoreCache.getSelfScore(exe.getMd5()); + count += 1; + } + catch (LSHException e) { + // Don't increment count + } + } + return count; + } + + @Override + public void resetStorage(double simThresh, double sigThresh) throws LSHException { + super.resetStorage(simThresh, sigThresh); + singleScore = null; + scoreCache.resetStorage(simThresh, sigThresh); + } + + @Override + public float getSingularSelfScore() { + return singleScore[singleExeXref - 1]; + } + + @Override + protected void initializeScores() { + // We only allocate the one row corresponding to our single, not the whole matrix + int size = executableSet.numExecutables(); + singleScore = new float[size]; + for (int i = 0; i < size; ++i) { + singleScore[i] = 0.0f; + } + } + + @Override + protected boolean checkPreliminaryPairThreshold(int hitcount, int pairThreshold) { + // Unless raw hit count exceeds threshold, + // delay decision until we can see the functions by returning true + return (hitcount < pairThreshold); + } + + @Override + protected void scorePair(FunctionPair pair) { + int indexA = pair.funcA.getExecutableRecord().getXrefIndex(); + int indexB = pair.funcB.getExecutableRecord().getXrefIndex(); + + if (indexA == singleExeXref) { + singleScore[indexB - 1] += pair.significance; + } + else if (indexB == singleExeXref) { + singleScore[indexA - 1] += pair.significance; + } + } + + @Override + protected List pairFunctions(LSHVectorFactory vectorFactory, + List vec2func, List vectors, int hitcount, + int pairThreshold) { + + // Separate cluster into functions that are in singleExe + // and functions that are NOT in singleExe (but are still in our filter set) + List singleFuncs = new ArrayList(hitcount); + List singleVec = new ArrayList(hitcount); + List otherFuncs = new ArrayList(hitcount); + List otherVec = new ArrayList(hitcount); + for (int i = 0; i < vec2func.size(); ++i) { + DescriptionManager manage = vec2func.get(i); + VectorResult curVec = vectors.get(i); + Iterator iter = manage.listAllFunctions(); + while (iter.hasNext()) { + FunctionDescription func = iter.next(); + int xrefIndex = func.getExecutableRecord().getXrefIndex(); + if (xrefIndex == 0) { + continue; // Not in our filter set + } + else if (xrefIndex == singleExeXref) { + singleFuncs.add(func); // Save off function in singleExe for later enumeration + singleVec.add(curVec); // Save off its associated vector + } + else { + otherFuncs.add(func); + otherVec.add(curVec); + } + } + } + + int pairCount = singleFuncs.size() * otherFuncs.size() + + (singleFuncs.size() * (singleFuncs.size() + 1)) / 2; + if (pairCount > pairThreshold) { + return null; + } + // For each function (funcA) in singleExe + // a) enumerate pairs with funcA and other functions in singleExe + // b) enumerate pairs with funcA and other functions NOT in singleExe + List pairs = new ArrayList(pairCount); + VectorCompare vectorCompare = new VectorCompare(); + for (int i = 0; i < singleFuncs.size(); ++i) { + FunctionDescription funcA = singleFuncs.get(i); + LSHVector vectorA = singleVec.get(i).vec; + LSHVector lastVec = vectorA; + double similarity = 1.0; + double significance = vectorFactory.getSelfSignificance(lastVec); + + // Create pairs from functions within singleExe + for (int j = i; j < singleFuncs.size(); ++j) { + LSHVector nextVec = singleVec.get(j).vec; + if (lastVec != nextVec) { + similarity = vectorA.compare(nextVec, vectorCompare); + significance = vectorFactory.calculateSignificance(vectorCompare); + lastVec = nextVec; + } + if (similarity < simThreshold || significance < sigThreshold) { + continue; + } + pairs.add(new FunctionPair(funcA, singleFuncs.get(j), similarity, significance)); + } + + // Create pairs with one function in singleExe and one function not in singleExe + for (int j = 0; j < otherFuncs.size(); ++j) { + LSHVector nextVec = otherVec.get(j).vec; + if (lastVec != nextVec) { + similarity = vectorA.compare(nextVec, vectorCompare); + significance = vectorFactory.calculateSignificance(vectorCompare); + lastVec = nextVec; + } + if (similarity < simThreshold || significance < sigThreshold) { + continue; + } + FunctionDescription funcB = otherFuncs.get(j); + pairs.add(new FunctionPair(funcA, funcB, similarity, significance)); + } + } + + return pairs; + } + + @Override + public float getScore(int a) { + return singleScore[a - 1]; + } + + @Override + public float getSelfScore(int a) throws LSHException { + if (a == singleExeXref) { + return singleScore[a - 1]; + } + ExecutableRecord executableRecord = index2ExeMap.get(a); + if (executableRecord == null) { + return 0.0f; + } + return scoreCache.getSelfScore(executableRecord.getMd5()); + } + + @Override + public void commitSelfScore() throws LSHException { + scoreCache.commitSelfScore(singleExe.getMd5(), singleScore[singleExeXref - 1]); + } + + @Override + protected void commitSelfScore(String md5, float selfScore) throws LSHException { + scoreCache.commitSelfScore(md5, selfScore); + } + + @Override + public float getScore(int a, int b) { + // We use only the single column score matrix + if (b == singleExeXref) { + return singleScore[a - 1]; + } + return singleScore[b - 1]; + + } + + /** + * Pre-load self-scores of the registered executables. + * @param missing (optional - may be null) will contain the list of exes missing a score + * @throws LSHException if there are problems loading scores + */ + public void prefetchSelfScores(List missing) + throws LSHException { + scoreCache.prefetchScores(executableSet.getExecutableRecordSet(), missing); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/FileScoreCaching.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/FileScoreCaching.java new file mode 100755 index 0000000000..96c80b109d --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/FileScoreCaching.java @@ -0,0 +1,169 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.io.*; +import java.util.*; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.ExecutableRecord; + +public class FileScoreCaching implements ScoreCaching { + + private File storageFile; // File holding the score data + private TreeMap cacheMap; // In memory cache of scores, indexed by md5 string + private double simThreshold; // similarity threshold loaded from file + private double sigThreshold; // significance threshold loaded from file + + public FileScoreCaching(String fileName) { + storageFile = new File(fileName); + cacheMap = null; + simThreshold = -1.0; // Negative indicates thresholds have not been configured + sigThreshold = -1.0; + } + + private void loadCache() throws IOException { + if (cacheMap != null) { + return; + } + if (!storageFile.exists()) { + return; // File doesn't exist, it will get created on first commitSelfScore call + } + cacheMap = new TreeMap(); + BufferedReader reader = new BufferedReader(new FileReader(storageFile)); + String floatString = reader.readLine(); + if (floatString == null) { + reader.close(); + throw new IOException("Score file missing threshold lines"); + } + simThreshold = Float.parseFloat(floatString); + floatString = reader.readLine(); + if (floatString == null) { + reader.close(); + throw new IOException("Score file missing threshold lines"); + } + sigThreshold = Float.parseFloat(floatString); + floatString = reader.readLine(); + while (floatString != null) { + String[] split = floatString.split(" "); + if (split.length != 2 || split[0].length() != 32) { + reader.close(); + throw new IOException("Bad line in score file"); + } + float val = Float.parseFloat(split[1]); + cacheMap.put(split[0], val); + floatString = reader.readLine(); + } + reader.close(); + } + + @Override + public float getSelfScore(String md5) throws LSHException { + try { + loadCache(); + } + catch (IOException e) { + throw new LSHException("Could not recover cached scores: " + e.getMessage()); + } + if (cacheMap != null) { + Float val = cacheMap.get(md5); + if (val != null) { + return val.floatValue(); + } + } + throw new LSHException("Self-score not recorded for " + md5); + } + + @Override + public void commitSelfScore(String md5, float score) throws LSHException { + try { + BufferedWriter writer = new BufferedWriter(new FileWriter(storageFile, true)); + if (cacheMap == null) { + writer.write(Double.toString(simThreshold)); + writer.newLine(); + writer.write(Double.toString(sigThreshold)); + writer.newLine(); + cacheMap = new TreeMap(); + } + cacheMap.put(md5, score); + writer.write(md5); + writer.append(' '); + writer.write(Float.toString(score)); + writer.newLine(); + writer.close(); + } + catch (IOException ex) { + throw new LSHException("Could not commit self-score: " + ex.getMessage()); + } + } + + @Override + public double getSimThreshold() throws LSHException { + try { + loadCache(); + } + catch (IOException e) { + throw new LSHException("Problems loading score cache: " + e.getMessage()); + } + return simThreshold; + } + + @Override + public double getSigThreshold() throws LSHException { + try { + loadCache(); + } + catch (IOException e) { + throw new LSHException("Problems loading score cache: " + e.getMessage()); + } + return sigThreshold; + } + + @Override + public void prefetchScores(Set exeSet, List missing) + throws LSHException { + try { + loadCache(); + } + catch (IOException e) { + throw new LSHException("Could not prefetch scores: " + e.getMessage()); + } + if (missing != null) { + if (cacheMap == null) { + for (ExecutableRecord exeRec : exeSet) { + missing.add(exeRec); // Everything is missing + } + } + else { + for (ExecutableRecord exeRec : exeSet) { + if (!cacheMap.containsKey(exeRec.getMd5())) { + missing.add(exeRec); + } + } + } + } + } + + @Override + public void resetStorage(double simThresh, double sigThresh) throws LSHException { + if (storageFile.exists()) { + storageFile.delete(); + } + cacheMap = null; + simThreshold = simThresh; + sigThreshold = sigThresh; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/FunctionDatabaseProxy.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/FunctionDatabaseProxy.java new file mode 100755 index 0000000000..4f972bc1f6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/FunctionDatabaseProxy.java @@ -0,0 +1,194 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.io.BufferedWriter; +import java.io.OutputStreamWriter; +import java.net.*; + +import org.xml.sax.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.protocol.*; +import ghidra.framework.client.ClientUtil; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +public class FunctionDatabaseProxy implements FunctionDatabase { + private DatabaseInformation info; + private LSHVectorFactory vectorFactory; + private URL httpURL; + private Error lasterror; + private Status status; + private boolean isinit; + private XmlErrorHandler xmlErrorHandler; + + static class XmlErrorHandler implements ErrorHandler { + + @Override + public void warning(SAXParseException exception) throws SAXException { + // Ignore warnings + } + + @Override + public void error(SAXParseException exception) throws SAXException { + throw exception; + } + + @Override + public void fatalError(SAXParseException exception) throws SAXException { + throw exception; + } + + } + + public FunctionDatabaseProxy(URL url) throws MalformedURLException { + httpURL = new URL(url.toString()); // Make sure URL has a real handler + lasterror = null; + info = null; + vectorFactory = FunctionDatabase.generateLSHVectorFactory(); + status = Status.Unconnected; + isinit = false; + xmlErrorHandler = new XmlErrorHandler(); + } + + @Override + public Status getStatus() { + return status; + } + + @Override + public ConnectionType getConnectionType() { + return ConnectionType.Unencrypted_No_Authentication; + } + + @Override + public String getUserName() { + return ClientUtil.getUserName(); + } + + @Override + public void setUserName(String userName) { + // Not currently implemented + } + + @Override + public LSHVectorFactory getLSHVectorFactory() { + return vectorFactory; + } + + @Override + public DatabaseInformation getInfo() { + return info; + } + + @Override + public int compareLayout() { + if (info.layout_version == PostgresFunctionDatabase.LAYOUT_VERSION) { + return 0; + } + return (info.layout_version < PostgresFunctionDatabase.LAYOUT_VERSION) ? -1 : 1; + } + + @Override + public String getURLString() { + return httpURL.toString(); + } + + @Override + public BSimServerInfo getServerInfo() { + return new BSimServerInfo(httpURL); + } + + @Override + public boolean initialize() { + if (isinit) { + return true; + } + if (httpURL == null) { + status = Status.Error; + lasterror = new FunctionDatabase.Error(ErrorCategory.Initialization, "MalformedURL"); + return false; + } + QueryInfo queryInfo = new QueryInfo(); + QueryResponseRecord response = query(queryInfo); + if (response == null) { + return false; + } + info = ((ResponseInfo) response).info; + status = Status.Ready; + isinit = true; + return true; + } + + @Override + public void close() { + status = Status.Unconnected; + isinit = false; + info = null; + } + + @Override + public Error getLastError() { + return lasterror; + } + + @Override + public QueryResponseRecord query(BSimQuery query) { + HttpURLConnection connection; + query.buildResponseTemplate(); + try { + lasterror = null; + connection = (HttpURLConnection) httpURL.openConnection(); + connection.setRequestMethod("POST"); + connection.setDoOutput(true); + BufferedWriter writer = + new BufferedWriter(new OutputStreamWriter(connection.getOutputStream())); + query.saveXml(writer); + writer.close(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(connection.getInputStream(), + "response", xmlErrorHandler, false); + if (parser.peek().getName().equals("error")) { + ResponseError respError = new ResponseError(); + respError.restoreXml(parser, vectorFactory); + parser.dispose(); + lasterror = new FunctionDatabase.Error(ErrorCategory.Fatal, respError.errorMessage); + query.clearResponse(); + return null; + } + QueryResponseRecord response = query.getResponse(); + response.restoreXml(parser, vectorFactory); + parser.dispose(); + if (response instanceof ResponseInfo) { + // Query is one of CreateDatabase, InstallCategoryRequest, InstallMetadataRequest, or QueryInfo + info = ((ResponseInfo) response).info; + status = Status.Ready; + isinit = true; + } + return response; + } + catch (Exception ex) { + lasterror = new FunctionDatabase.Error(ErrorCategory.Connection, ex.getMessage()); + status = Status.Error; + query.clearResponse(); + return null; + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/IDSQLResolution.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/IDSQLResolution.java new file mode 100755 index 0000000000..fee1db3d0e --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/IDSQLResolution.java @@ -0,0 +1,111 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.sql.SQLException; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.ExecutableRecord; + +/** + * Class for managing filter elements (FilterTemplate) that need to be resolved (typically to an id) + * before they can be converted to an SQL clause. + * + */ +public abstract class IDSQLResolution { + public long id1; // First resolved id + public long id2; // Second resolved id + + public abstract void resolve(AbstractSQLFunctionDatabase columnDatabase, + ExecutableRecord exe) throws SQLException; + + public static class Architecture extends IDSQLResolution { // Architecture string + private String archName; // Architecture name as a string + + public Architecture(String nm) { + archName = nm; + id1 = 0; + } + + @Override + public void resolve(AbstractSQLFunctionDatabase columnDatabase, ExecutableRecord exe) + throws SQLException { + if (id1 == 0) + id1 = columnDatabase.queryArchString(archName); + } + } + + public static class Compiler extends IDSQLResolution { + private String compilerName; // Compiler name as a string + + public Compiler(String nm) { + compilerName = nm; + id1 = 0; + } + + @Override + public void resolve(AbstractSQLFunctionDatabase columnDatabase, ExecutableRecord exe) + throws SQLException { + if (id1 == 0) + id1 = columnDatabase.queryCompilerString(compilerName); + } + } + + public static class ExeCategory extends IDSQLResolution { + private String categoryString; // Name of category as a string + private String valueString; // Value of category as a string + + public ExeCategory(String cat, String val) { + categoryString = cat; + valueString = val; + id1 = 0; + id2 = 0; + } + + @Override + public void resolve(AbstractSQLFunctionDatabase columnDatabase, ExecutableRecord exe) + throws SQLException { + if (id1 == 0) { + id1 = columnDatabase.queryCategoryString(categoryString); + id2 = columnDatabase.queryCategoryString(valueString); + } + } + } + + public static class ExternalFunction extends IDSQLResolution { + private String exeName; // Name of executable containing external function + private String funcName; // Name of external function + + public ExternalFunction(String exe, String func) { + exeName = exe; + funcName = func; + id1 = 0; + } + + @Override + public void resolve(AbstractSQLFunctionDatabase columnDatabase, ExecutableRecord exe) + throws SQLException { + try { + if (id1 == 0) + id1 = columnDatabase.recoverExternalFunctionId(exeName, funcName, + exe.getArchitecture()); + } + catch (LSHException ex) { + throw new SQLException(ex.getMessage()); + } + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/IdHistogram.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/IdHistogram.java new file mode 100755 index 0000000000..aa3dce9d84 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/IdHistogram.java @@ -0,0 +1,96 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.util.*; + +import generic.lsh.vector.LSHVector; +import ghidra.features.bsim.query.description.*; + +/** + * Lightweight object container of an LSHVector and its count within a collection of functions (database/executable) + * TODO: This should likely be merged with SignatureRecord + */ +public class IdHistogram implements Comparable { + public long id; // Is the unique id of the vector as computed by LSHVector.getVectorId() + public int count; // Is the count of duplicate vectors within the larger set of functions + public LSHVector vec = null; // Is an instance of the vector itself + + @Override + public int compareTo(IdHistogram o) { + return Long.compare(id, o.id); + } + + /** + * @param iter is iterator over functions whose vectors are to be histogrammed + * @return the sorted list of pairs (hash,count) + */ + public static TreeSet buildVectorIdHistogram(Iterator iter) { + TreeSet table = new TreeSet(); + IdHistogram testItem = new IdHistogram(); + while(iter.hasNext()) { + testItem.id = iter.next().getVectorId(); + if (testItem.id == 0) { + continue; // Function doesn't have associated vector + } + IdHistogram cur = table.floor(testItem); + if (cur == null || cur.id != testItem.id) { + cur = new IdHistogram(); + cur.id = testItem.id; + cur.count = 1; + table.add(cur); + } + else { + cur.count += 1; + } + } + return table; + } + + /** + * Organize/histogram LSHVectors by hash. Take into account functions that don't have a vector. + * Record hashes in the FunctionDescription's SignatureRecord + * @param manage is the container of the FunctionDescriptions + * @param iter is the iterator over the FunctionDescriptions being collected + * @return the histogram as a set of (id,count,vec) triples + */ + public static Set collectVectors(DescriptionManager manage,Iterator iter) { + TreeSet res = new TreeSet(); + IdHistogram workingRec = new IdHistogram(); + while(iter.hasNext()) { + FunctionDescription desc = iter.next(); + SignatureRecord sigrec = desc.getSignatureRecord(); + if (sigrec == null) { + continue; + } + long key = sigrec.getLSHVector().calcUniqueHash(); + manage.setSignatureId(sigrec, key); + workingRec.id = key; + IdHistogram prevRec = res.floor(workingRec); + if (prevRec == null || prevRec.id != key) { + prevRec = new IdHistogram(); + prevRec.id = key; + prevRec.vec = sigrec.getLSHVector(); + prevRec.count = 1; + res.add(prevRec); + } + else { + prevRec.count += 1; + } + } + return res; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/NoDatabaseException.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/NoDatabaseException.java new file mode 100755 index 0000000000..e06855762c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/NoDatabaseException.java @@ -0,0 +1,24 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +public class NoDatabaseException extends Exception { + private static final long serialVersionUID = 1L; + + public NoDatabaseException(String msg) { + super(msg); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/PostgresFunctionDatabase.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/PostgresFunctionDatabase.java new file mode 100755 index 0000000000..ce23733cb1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/PostgresFunctionDatabase.java @@ -0,0 +1,579 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.io.IOException; +import java.net.URL; +import java.sql.*; +import java.util.*; +import java.util.logging.Level; +import java.util.logging.Logger; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.WeightedLSHCosineVectorFactory; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.BSimPostgresDBConnectionManager.BSimPostgresDataSource; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.client.tables.CachedStatement; +import ghidra.features.bsim.query.client.tables.SQLStringTable; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.*; + +/** + * Defines the BSim {@link FunctionDatabase} backed by a PostgreSQL database. + * + * Simple, one-column tables that only contain string data use the + * {@link SQLStringTable} class and are defined in this class. More complex + * tables are defined in their own classes in the + * {@link ghidra.features.bsim.query.client.tables} package. + * + */ +public final class PostgresFunctionDatabase + extends AbstractSQLFunctionDatabase { + + // NOTE: Previously named ColumnDatabase + + static { + // FIXME: Decide how and where logging should be established + // FIXME: Logging should be disabled by default + Logger postgresLogger = Logger.getLogger("org.postgresql.Driver"); + postgresLogger.setLevel(Level.FINEST); + } + + // Indicates the version of the db table configuration. This needs to be updated + // whenever changes are made to the table structure. + public static final int LAYOUT_VERSION = 6; + + private static final String DEFAULT_DATABASE_NAME = "postgres"; + + private BSimPostgresDataSource postgresDs; + private boolean asynchronous; // Should database commits be asynchronous + + // Persist SQL statements as class members so we don't have to recreate them every + // time they're needed. + private final CachedStatement reusableStatement = new CachedStatement<>(); + private final CachedStatement selectVectorByRowIdStatement = + new CachedStatement<>(); + private final CachedStatement selectNearestVectorStatement = + new CachedStatement<>(); + + public PostgresFunctionDatabase(URL postgresUrl, boolean async) { + super(BSimPostgresDBConnectionManager.getDataSource(postgresUrl), + FunctionDatabase.generateLSHVectorFactory(), LAYOUT_VERSION); + postgresDs = (BSimPostgresDataSource) ds; + asynchronous = async; + } + + @Override + public void close() { + reusableStatement.close(); + selectVectorByRowIdStatement.close(); + selectNearestVectorStatement.close(); + super.close(); + } + + private Statement getReusableStatement() throws SQLException { + return reusableStatement.prepareIfNeeded(() -> initConnection().createStatement()); + } + + /** + * Obtain an exclusive lock on the main tables. To be used when INSERTING, DELETING, or UPDATING + * to prevent concurrent changes to the tables. + * @throws SQLException if the server reports an error + */ + @Override + protected void lockTablesForWrite() throws SQLException { + String stmtstring = "LOCK TABLE exetable, desctable, vectable IN SHARE ROW EXCLUSIVE MODE"; + getReusableStatement().execute(stmtstring); + } + + private void changePassword(Connection c, String username, char[] newPassword) + throws SQLException { + StringBuilder buffer = new StringBuilder(); + buffer.append("ALTER ROLE \""); + buffer.append(username); + buffer.append("\" WITH PASSWORD '"); + for (char ch : newPassword) { + if (ch == '\'') { + buffer.append(ch); // Escape single quote by appending it twice + } + buffer.append(ch); + } + buffer.append('\''); + // Don't think jdbc does anything to this statement to encrypt password before sending it. + // The connection with the server SHOULD be under SSL at this point + try (Statement st = c.createStatement()) { + + st.executeUpdate(buffer.toString()); + + // update password to be used for new connections + postgresDs.setPassword(username, newPassword); + } + } + + /** + * + * @param st the database query statement + * @throws SQLException if there is a problem creating the query statement + */ + private void createVectorFunctions(Statement st) throws SQLException { + st.executeUpdate("CREATE FUNCTION insert_vec(newvec lshvector,OUT ourhash BIGINT) AS $$" + + " DECLARE" + + " curs1 CURSOR (key BIGINT) FOR SELECT count FROM vectable WHERE id = key FOR UPDATE;" + + " ourcount INTEGER;" + " BEGIN" + " ourhash := lshvector_hash(newvec);" + + " OPEN curs1( ourhash );" + " FETCH curs1 INTO ourcount;" + " IF FOUND THEN" + + " UPDATE vectable SET count = ourcount + 1 WHERE CURRENT OF curs1;" + " ELSE" + + " INSERT INTO vectable (id,count,vec) VALUES(ourhash,1,newvec);" + " END IF;" + + " CLOSE curs1;" + " END;" + " $$ LANGUAGE plpgsql;"); + st.executeUpdate( + "CREATE FUNCTION remove_vec(vecid BIGINT,countdiff INTEGER) RETURNS INTEGER AS $$" + + "DECLARE" + + " curs1 CURSOR (key BIGINT) FOR SELECT count FROM vectable WHERE id = key FOR UPDATE;" + + " ourcount INTEGER;" + " rescode INTEGER;" + "BEGIN" + " rescode = -1;" + + " OPEN curs1( vecid );" + " FETCH curs1 INTO ourcount;" + + " IF FOUND AND ourcount > countdiff THEN" + + " UPDATE vectable SET count = ourcount - countdiff WHERE CURRENT OF curs1;" + + " rescode = 0;" + " ELSIF FOUND THEN" + + " DELETE FROM vectable WHERE CURRENT OF curs1;" + " rescode = 1;" + + " END IF;" + " CLOSE curs1;" + " RETURN rescode;" + "END;" + + "$$ LANGUAGE plpgsql;"); + } + + /** + * + * @throws SQLException if there is an error creating/executing the query + */ + private void serverLoadWeights(Connection db) throws SQLException { + try (Statement st = db.createStatement(); + ResultSet rs = st.executeQuery("SELECT lsh_load()")) { + while (rs.next()) { + // int val = rs.getInt(1); + } + } + } + + @Override + protected void initializeDatabase(Configuration config) throws SQLException { + + Connection db = initConnection(); + serverLoadWeights(db); + + if (asynchronous) { + try (Statement st = db.createStatement()) { + // Tell server to do asynchronous commits. This speeds up large + // ingests with a (slight) danger of + // losing the most recent commits if the server crashes (NOTE: + // database integrity should still be recoverable) + st.executeUpdate("SET LOCAL synchronous_commit TO OFF"); + } + } + + super.initializeDatabase(config); + } + + @Override + protected void generateRawDatabase() throws SQLException { + BSimServerInfo serverInfo = postgresDs.getServerInfo(); + BSimServerInfo defaultServerInfo = new BSimServerInfo(DBType.postgres, + serverInfo.getServerName(), serverInfo.getPort(), DEFAULT_DATABASE_NAME); + String createdbstring = "CREATE DATABASE \"" + serverInfo.getDBName() + '"'; + BSimPostgresDataSource defaultDs = + BSimPostgresDBConnectionManager.getDataSource(defaultServerInfo); + try (Connection db = defaultDs.getConnection(); Statement st = db.createStatement()) { + st.executeUpdate(createdbstring); + postgresDs.initializeFrom(defaultDs); + } + finally { + defaultDs.dispose(); + } + } + + @Override + protected void createDatabase(Configuration config) throws SQLException { + try { + super.createDatabase(config); + + Connection db = super.initConnection(); + try (Statement st = db.createStatement()) { + + st.executeUpdate("CREATE EXTENSION IF NOT EXISTS lshvector"); + st.executeUpdate( + "CREATE TABLE vectable(id BIGINT UNIQUE,count INTEGER,vec lshvector)"); + st.executeUpdate( + "CREATE INDEX vectable_vec_idx ON vectable USING gin (vec gin_lshvector_ops)"); + createVectorFunctions(st); + + st.executeUpdate("REVOKE ALL ON SCHEMA PUBLIC FROM PUBLIC"); + st.executeUpdate("GRANT USAGE ON SCHEMA PUBLIC TO PUBLIC"); + st.executeUpdate("GRANT SELECT ON ALL TABLES IN SCHEMA PUBLIC TO PUBLIC"); + st.executeUpdate("GRANT USAGE ON ALL SEQUENCES IN SCHEMA PUBLIC TO PUBLIC"); + + serverLoadWeights(db); + + if (asynchronous) { + // Tell server to do asynchronous commits. This speeds up large + // ingests with a (slight) danger of + // losing the most recent commits if the server crashes (NOTE: + // database integrity should still be recoverable) + st.executeUpdate("SET LOCAL synchronous_commit TO OFF"); + } + } + } + catch (final SQLException err) { + throw new SQLException("Could not create database: " + err.getMessage()); + } + } + + /** + * + * @throws SQLException if there is a problem creating or executing the query + */ + private void dropIndex(Connection c) throws SQLException { + try (Statement st = c.createStatement()) { + st.execute("DROP INDEX vectable_vec_idx"); + } + } + + /** + * + * @throws SQLException if there is a problem creating or executing the query + */ + private void rebuildIndex(Connection c) throws SQLException { + try (Statement st = c.createStatement(); + ResultSet rs = st.executeQuery("SELECT lsh_reload()")) { + st.execute("SET maintenance_work_mem TO '2GB'"); + st.execute( + "CREATE INDEX vectable_vec_idx ON vectable USING gin (vec gin_lshvector_ops)"); + } + } + + /** + * Attempt to preload some of the main tables and indices into cache or RAM + * + * @param mainIndex + * For the main index -- 0=don't load 1=load into RAM 2=load into + * cache + * @param secondaryIndex + * For the secondary index -- 0=don't load 1=load into RAM 2=load + * into cache + * @param vectors + * For vectors -- 0=don't load 1=load into RAM 2=load into cache + * @return the number of blocks loaded for the main index + * @throws SQLException if there is a problem creating or executing the query + */ + private int preWarm(Connection c, int mainIndex, int secondaryIndex, int vectors) + throws SQLException { + try (Statement st = c.createStatement()) { + // Try to load the entire main index into the PostgreSQL cache + int res = -1; + String queryString; + st.execute("CREATE EXTENSION IF NOT EXISTS pg_prewarm"); + if (mainIndex != 0) { + if (mainIndex == 1) { + queryString = "SELECT pg_prewarm('vectable_vec_idx','read')"; + } + else { + queryString = "SELECT pg_prewarm('vectable_vec_idx')"; + } + try (ResultSet rs = st.executeQuery(queryString)) { + if (rs.next()) { + res = rs.getInt(1); // Number of blocks that were successfully prewarmed + while (rs.next()) { // Shouldn't be any more rows + // TODO: Should be no need to exhaust results + } + } + } + } + if (secondaryIndex != 0) { + // Try to convince the OS to load the secondary vector table index + // into RAM + if (secondaryIndex == 1) { + queryString = "SELECT pg_prewarm('vectable_id_key','read')"; + } + else { + queryString = "SELECT pg_prewarm('vectable_id_key')"; + } + try (ResultSet rs = st.executeQuery(queryString)) { + while (rs.next()) { + // TODO: Should be no need to exhaust results + // int val = rs.getInt(1); + } + } + } + if (vectors != 0) { + // Try to convince the OS to load the vector table into RAM + if (vectors == 1) { + queryString = "SELECT pg_prewarm('vectable','read')"; + } + else { + queryString = "SELECT pg_prewarm('vectable')"; + } + try (ResultSet rs = st.executeQuery(queryString)) { + while (rs.next()) { + // TODO: Should be no need to exhaust results + // int val = rs.getInt(1); + } + } + } + st.execute("DROP EXTENSION pg_prewarm"); + + return res; + } + } + + /** + * Make sure the vector corresponding to the SignatureRecord is inserted into the vectable + * @param sigrec is the SignatureRecord + * @return the computed id of the vector + * @throws SQLException if there is a problem creating or executing the query + */ + @Override + protected long storeSignatureRecord(SignatureRecord sigrec) throws SQLException { + String sql = "SELECT insert_vec( '" + sigrec.getLSHVector().saveSQL() + "')"; + try (ResultSet rs = getReusableStatement().executeQuery(sql)) { + if (!rs.next()) { + throw new SQLException("Did not get vector id after insertion"); + } + return rs.getLong(1); + } + } + + /** + * Low level count decrement of a vector record from vectable, if count + * reaches zero, the record is deleted + * @param c database connection + * @param id vector row ID + * @param countdiff the amount to subtract from count + * @return 0 if decrement short of 0, return 1 if record was removed, return + * -1 if there was a problem + * @throws SQLException if there is a problem creating or executing the query + */ + @Override + protected int deleteVectors(long id, int countdiff) throws SQLException { + int res = -100; + String sql = + "SELECT remove_vec( " + Long.toString(id) + ',' + Integer.toString(countdiff) + ")"; + try (ResultSet rs = getReusableStatement().executeQuery(sql)) { + if (!rs.next()) { + throw new SQLException("Did not get result code after deletion"); + } + res = rs.getInt(1); + } + return res; + } + + /** + * + * @param resultset the list of result set objects to populate + * @param vec the vector containing the saveSQL query statement + * @param simthresh the similarity threshold + * @param sigthresh the confidence threshold + * @param max the max number of results to return + * @return the number of results returned + * @throws SQLException if there is a problem creating or executing the query + */ + @Override + protected int queryNearestVector(List resultset, LSHVector vec, + double simthresh, double sigthresh, int max) throws SQLException { + PreparedStatement s = + selectNearestVectorStatement.prepareIfNeeded(() -> initConnection().prepareStatement( + "WITH const(cvec) AS (VALUES( lshvector_in( CAST( ? AS cstring) ) ) )," + + " comp AS (" + + " SELECT id,count,cvec,vec,lshvector_compare(cvec,vec) AS cfunc FROM const,vectable" + + " WHERE cvec % vec)" + + " SELECT id,count,(comp.cfunc).sim,(comp.cfunc).sig,vec FROM comp" + + " WHERE (comp.cfunc).sim > ? AND (comp.cfunc).sig > ?" + + " ORDER BY (comp.cfunc).sim DESC" + " LIMIT ?")); + s.setString(1, vec.saveSQL()); + s.setDouble(2, simthresh); + s.setDouble(3, sigthresh); + s.setInt(4, max); + + int total = 0; + try (ResultSet rs = s.executeQuery()) { + while (rs.next()) { + VectorResult curres = new VectorResult(); + resultset.add(curres); + curres.vectorid = rs.getLong(1); + curres.hitcount = rs.getInt(2); + curres.sim = rs.getDouble(3); + curres.signif = rs.getDouble(4); + final String vecstring = rs.getString(5); + try { + curres.vec = vectorFactory.restoreVectorFromSql(vecstring); + } + catch (final IOException e) { + throw new SQLException(e.getMessage()); + } + total += curres.hitcount; + } + return total; + } + } + + @Override + protected void queryNearestVector(QueryNearestVector query) throws SQLException { + ResponseNearestVector response = query.nearresponse; + response.totalvec = 0; + response.totalmatch = 0; + response.uniquematch = 0; + + int vectormax = query.vectormax; + if (vectormax == 0) { + vectormax = 2000000; // Really means a very big limit + } + + Iterator iter = query.manage.listAllFunctions(); + while (iter.hasNext()) { + FunctionDescription frec = iter.next(); + SignatureRecord srec = frec.getSignatureRecord(); + if (srec == null) { + continue; + } + LSHVector thevec = srec.getLSHVector(); + double len2 = vectorFactory.getSelfSignificance(thevec); + if (len2 < query.signifthresh) { + continue; + } + + response.totalvec += 1; + List resultset = new ArrayList<>(); + + queryNearestVector(resultset, thevec, query.thresh, query.signifthresh, vectormax); + if (resultset.isEmpty()) { + continue; + } + SimilarityVectorResult simres = new SimilarityVectorResult(frec); + simres.addNotes(resultset); + response.totalmatch += simres.getTotalCount(); + if (simres.getTotalCount() == 1) { + response.uniquematch += 1; + } + response.result.add(simres); + } + } + + @Override + protected VectorResult queryVectorId(long id) throws SQLException { + PreparedStatement s = selectVectorByRowIdStatement.prepareIfNeeded(() -> initConnection() + .prepareStatement("SELECT id,count,vec FROM vectable WHERE id = ?")); + s.setLong(1, id); + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { + throw new SQLException("Bad vectable rowid"); + } + VectorResult rowres; + try { + rowres = new VectorResult(); + rowres.vectorid = rs.getLong(1); + rowres.hitcount = rs.getInt(2); + rowres.vec = vectorFactory.restoreVectorFromSql(rs.getString(3)); + } + catch (final IOException e) { + throw new SQLException(e.getMessage()); + } + + return rowres; + } + + } + + @Override + public String getUserName() { + return postgresDs.getUserName(); + } + + @Override + public void setUserName(String userName) { + if (postgresDs.getStatus() == Status.Ready) { + throw new IllegalStateException("Connection has already been established"); + } + postgresDs.setPreferredUserName(userName); + } + + @Override + public QueryResponseRecord doQuery(BSimQuery query, Connection c) + throws SQLException, LSHException, DatabaseNonFatalException { + + if (query instanceof PrewarmRequest q) { + fdbPrewarm(q, c); + } + else if (query instanceof PasswordChange q) { + fdbPasswordChange(q, c); + } + else if (query instanceof AdjustVectorIndex q) { + fdbAdjustVectorIndex(q, c); + } + else { + return super.doQuery(query, c); + } + return query.getResponse(); + } + + /** + * Entry point for the AdjustVectorIndex command + * @param query the query to execute + * @throws SQLException if there is a problem rebuilding or dropping the index + */ + private void fdbAdjustVectorIndex(AdjustVectorIndex query, Connection c) throws SQLException { + ResponseAdjustIndex response = query.adjustresponse; + response.success = false; + if (query.doRebuild) { + rebuildIndex(c); + } + else { + dropIndex(c); + } + response.success = true; + } + + /** + * Entry point for the PrewarmRequest command + * @param request the prewarm request + * @param c Postgres DB connection + * @throws SQLException if there is an error issuing the query + */ + private void fdbPrewarm(PrewarmRequest request, Connection c) throws SQLException { + ResponsePrewarm response = request.prewarmresponse; + response.blockCount = preWarm(c, request.mainIndexConfig, request.secondaryIndexConfig, + request.vectorTableConfig); + } + + private void fdbPasswordChange(PasswordChange query, Connection c) throws LSHException { + ResponsePassword response = query.passwordResponse; + if (query.username == null) { + throw new LSHException("Missing username for password change"); + } + if (query.newPassword == null || query.newPassword.length == 0) { + throw new LSHException("No password provided"); + } + response.changeSuccessful = true; // Response parameters assuming success + response.errorMessage = null; + try { + changePassword(c, query.username, query.newPassword); + } + catch (SQLException e) { + response.changeSuccessful = false; + response.errorMessage = e.getMessage(); + } + } + + @Override + public String formatBitAndSQL(String v1, String v2) { + return "(" + v1 + " & " + v2 + ")"; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/RowKeySQL.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/RowKeySQL.java new file mode 100755 index 0000000000..f9454eface --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/RowKeySQL.java @@ -0,0 +1,50 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import ghidra.features.bsim.query.description.RowKey; + +public class RowKeySQL extends RowKey { + private long id; // Unique row id for the record + + public RowKeySQL(long i) { + id = i; + } + + @Override + public int compareTo(RowKey obj) { + RowKeySQL o = (RowKeySQL)obj; + return Long.compare(id, o.id); + } + + @Override + public boolean equals(Object obj) { + if (obj == null) + return false; + RowKeySQL o = (RowKeySQL)obj; + return id == o.id; + } + + @Override + public int hashCode() { + return Long.hashCode(id); + } + + @Override + public long getLong() { + return id; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/SQLEffects.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/SQLEffects.java new file mode 100755 index 0000000000..77c9f35b24 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/SQLEffects.java @@ -0,0 +1,185 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.sql.SQLException; +import java.util.*; +import java.util.Map.Entry; + +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.query.SQLFunctionDatabase; +import ghidra.features.bsim.query.protocol.BSimFilter; +import ghidra.features.bsim.query.protocol.FilterAtom; + +/** + * Container for collecting and sorting SQL string representations of FilterTemplates + * + */ +public class SQLEffects { + private boolean exetable = false; // true if the filter needs to reference (join) against the exetable + private boolean pathtable = false; // true if the filter needs to reference (join) against the pathtable + private int filterMask = 0; // Each 1-bit represents a single function tag that needs to be matched + private int filterValue = 0; // With the filterMask, bits indicate whether an individual + // function tag should match as true (1) or false(0) + + // Collection of SQL string pieces, sorted by the FilterTemplate that created them + private Map> wherelist = + new TreeMap>(); + + // Collection of SQL string pieces holding a join expression or the final function tag expression + private List linkClauses = new ArrayList(); + +// /** +// * Container for final SQL string sections; +// * +// */ +// public static class Cache { +// public String tableclause; +// public String whereclause; +// } + + public void setExeTable() { + exetable = true; + } + + public void setPathTable() { + pathtable = true; + } + + public void addFunctionFilter(int flag, boolean val) { + filterMask |= flag; // Check the specific bit + if (val) { + filterValue |= flag; // must be set to 1 + } + } + + /** + * Generate the string pieces of the WHERE clause, based on the FilterAtoms within the general filter, + * sort them into the wherelist container + * @param exefilter is the general filter + * @param idres is an array of precalculated ids associated with each FilterAtom + * @throws SQLException for errors building the SQL clause + */ + private void generateWhereClause(BSimFilter exefilter, IDSQLResolution idres[], + SQLFunctionDatabase db) throws SQLException { + + for (int i = 0; i < exefilter.numAtoms(); ++i) { + FilterAtom atom = exefilter.getAtom(i); + atom.type.gatherSQLEffect(this, atom, idres[i]); + } + + if (filterMask != 0) { + StringBuilder buf = new StringBuilder(); + String maskedFlags = + db.formatBitAndSQL("desctable.flags", Integer.toString(filterMask)); + buf.append(maskedFlags).append(" = ").append(filterValue); + addLink(buf.toString()); + } + + if (exetable) { + addLink("desctable.id_exe = exetable.id"); + } + if (pathtable) { + addLink("exetable.path = pathtable.id"); + } + } + + /** + * Given our sorted container of string pieces, combine them into a single SQL where clause, + * connecting them appropriately with 'AND' and 'OR' keywords and parentheses. + * @return the final where String + */ + private String buildWhereClause() { + StringBuilder builder = new StringBuilder(); + if (!linkClauses.isEmpty()) { + builder.append(" AND ("); + boolean printAnd = false; + for (String link : linkClauses) { + if (printAnd) { + builder.append(" AND "); + } + builder.append(link); + printAnd = true; + } + builder.append(')'); + } + + for (Entry> entry : wherelist.entrySet()) { + // Start with an AND clause because there are other clauses in the SQL string that may + // have been added before this - even if there aren't, having this here will not cause + // a problem. + builder.append(" AND "); + + BSimFilterType filter = entry.getKey(); + String finalClause = filter.buildSQLCombinedClause(entry.getValue()); + builder.append(finalClause); + } + + return builder.toString(); + } + + private String buildTableClause() { + StringBuilder buf = new StringBuilder(); + if (exetable) { + buf.append(",exetable"); + } + if (pathtable) { + buf.append(",pathtable"); + } + return buf.toString(); + } + + public void addLink(String value) { + linkClauses.add(value); + } + + public void addWhere(BSimFilterType filter, String val) { + List list = wherelist.get(filter); + if (list == null) { + list = new ArrayList(); + wherelist.put(filter, list); + } + list.add(val); + } + + /** + * Given a general ExecutableFilter object, return a set of matching SQL string pieces, + * ready to be pasted into the full SQL statement. The routine is handed an array of IDResolution references + * matching individual FilterAtoms as returned by ExecutableFilter.getAtom(i). The IDResolution, if non-null, + * holds any pre-calculated ids associated with the corresponding FilterAtom + * @param exeFilter is the general filter + * @param idres is the array holding pre-calculated ids + * @param db SQL function database + * @return BSimFilterSQL, holding the table clause and the where clause + * @throws SQLException for errors building the SQL clause + */ + public static BSimSqlClause createFilter(BSimFilter exeFilter, IDSQLResolution idres[], + SQLFunctionDatabase db) throws SQLException { + String tableclause = null; + String whereclause = null; + + SQLEffects effects = new SQLEffects(); + effects.generateWhereClause(exeFilter, idres, db); + whereclause = effects.buildWhereClause(); + if ((whereclause == null) || (whereclause.length() == 0)) { + return null; + } + + tableclause = effects.buildTableClause(); + + return new BSimSqlClause(tableclause, whereclause); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ScoreCaching.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ScoreCaching.java new file mode 100755 index 0000000000..b7c183c7be --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/ScoreCaching.java @@ -0,0 +1,78 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.util.List; +import java.util.Set; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.ExecutableRecord; + +/** + * Store and retrieve self-significance scores for executables specified by md5. + * These are generally too expensive to compute on the fly, so this class + * provides a persistence model for obtaining them. These scores also depend + * on specific threshold settings, so there is a method for checking the settings. + */ +public interface ScoreCaching { + + /** + * Pre-load self-scores for a set of executables. + * @param exeSet is the set of executables to check + * @param missing (optional - may be null) will contain the list of exes missing a score + * @throws LSHException if there are problems loading scores + */ + public void prefetchScores(Set exeSet, List missing) + throws LSHException; + + /** + * Retrieve the self-significance score for a given executable + * @param md5 is the 32-character md5 string specifying the executable + * @return the corresponding score + * @throws LSHException if the score is not obtainable + */ + public float getSelfScore(String md5) throws LSHException; + + /** + * Commit a new self-significance score for an executable + * @param md5 is the 32-character md5 string specifying the executable + * @param score is the score to commit + * @throws LSHException if there's a problem saving the value + */ + public void commitSelfScore(String md5, float score) throws LSHException; + + /** + * @return similarity threshold configured with this cache + * OR return -1 if the score is unconfigured + * @throws LSHException for problems retrieving configuration + */ + public double getSimThreshold() throws LSHException; + + /** + * @return significance threshold configured with this cache + * OR return -1 if the score is unconfigured + * @throws LSHException for problems retrieving configuration + */ + public double getSigThreshold() throws LSHException; + + /** + * Clear out any existing scores, and reset to an empty database + * @param simThresh is new similarity threshold to associate with scores + * @param sigThresh is new significance threshold to associate with scores + * @throws LSHException if there is a problem modifying storage + */ + public void resetStorage(double simThresh, double sigThresh) throws LSHException; +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/TableScoreCaching.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/TableScoreCaching.java new file mode 100755 index 0000000000..16ea74c433 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/TableScoreCaching.java @@ -0,0 +1,223 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.sql.Types; +import java.util.*; + +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.protocol.*; + +public class TableScoreCaching implements ScoreCaching { + + private final static String TABLE_NAME = "exeselfscores"; + private final static String SIMILARITY_KEY = "similarity"; + private final static String SIGNIFICANCE_KEY = "significance"; + private final static int VALUES_PER_QUERY = 100; + private FunctionDatabase db; // Active connection to the database + private TreeMap cacheMap; // In memory cache of scores, indexed by md5 string + private double simThreshold; // similarity threshold loaded from file + private double sigThreshold; // significance threshold loaded from file + private QueryOptionalValues queryValue; // Template for querying scores + private InsertOptionalValues insertValue; // Template for inserting scores + + public TableScoreCaching(FunctionDatabase d) { + db = d; + cacheMap = null; + simThreshold = -1.0; + sigThreshold = -1.0; + queryValue = new QueryOptionalValues(); + queryValue.tableName = TABLE_NAME; + queryValue.keyType = Types.VARCHAR; + queryValue.valueType = Types.REAL; + queryValue.keys = new Object[1]; + insertValue = new InsertOptionalValues(); + insertValue.tableName = TABLE_NAME; + insertValue.keyType = Types.VARCHAR; + insertValue.valueType = Types.REAL; + insertValue.keys = new Object[1]; + insertValue.values = new Object[1]; + } + + /** + * Do minimal work to set up query template for a given number of values + * @param size is the number of values to be queried + */ + private void setUpQuery(int size) { + if (queryValue != null && queryValue.keys.length == size) { + return; + } + queryValue.keys = new Object[size]; + } + + /** + * Do minimal work to set up insert template for a given number of values + * @param size is the number of values to be inserted + */ + private void setUpInsert(int size) { + if (insertValue != null && insertValue.keys.length == size) { + return; + } + insertValue.keys = new Object[size]; + insertValue.values = new Object[size]; + } + + /** + * Make sure the backing database table exists, and if it doesn't, create it. + * If the table existed previously, try to read thresholds from it + * @throws LSHException for problems with the connection + */ + private void initialize() throws LSHException { + if (cacheMap != null) { + return; + } + cacheMap = new TreeMap(); + QueryOptionalExist query = new QueryOptionalExist(); + query.tableName = TABLE_NAME; + query.keyType = Types.VARCHAR; + query.valueType = Types.REAL; + query.attemptCreation = true; // Create table if it doesn't already exist + ResponseOptionalExist response = query.execute(db); + if (response == null) { + throw new LSHException(db.getLastError().message); + } + if (response.wasCreated) { + return; + } + setUpQuery(2); + queryValue.keys[0] = SIMILARITY_KEY; + queryValue.keys[1] = SIGNIFICANCE_KEY; + ResponseOptionalValues optionalresponse = queryValue.execute(db); + if (optionalresponse == null) { + throw new LSHException(db.getLastError().message); + } + Float simObj = (Float) optionalresponse.resultArray[0]; + Float sigObj = (Float) optionalresponse.resultArray[1]; + if (simObj != null && sigObj != null) { + simThreshold = simObj.doubleValue(); + sigThreshold = sigObj.doubleValue(); + } + } + + @Override + public void prefetchScores(Set exeSet, List missing) + throws LSHException { + initialize(); + int size = exeSet.size(); + Iterator iter = exeSet.iterator(); + ExecutableRecord[] queryGroup = new ExecutableRecord[VALUES_PER_QUERY]; + while (size > 0) { + int curSize = size > VALUES_PER_QUERY ? VALUES_PER_QUERY : size; + setUpQuery(curSize); + for (int i = 0; i < curSize; ++i) { + queryGroup[i] = iter.next(); + queryValue.keys[i] = queryGroup[i].getMd5(); + } + ResponseOptionalValues response = queryValue.execute(db); + if (response == null) { + throw new LSHException(db.getLastError().message); + } + Object[] result = response.resultArray; + for (int i = 0; i < curSize; ++i) { + if (result[i] != null) { + cacheMap.put((String) queryValue.keys[i], (Float) result[i]); + } + else if (missing != null) { + missing.add(queryGroup[i]); + } + } + size -= curSize; + } + } + + @Override + public float getSelfScore(String md5) throws LSHException { + initialize(); + Float val = cacheMap.get(md5); + if (val != null) { + return val.floatValue(); + } + setUpQuery(1); + queryValue.keys[0] = md5; + ResponseOptionalValues response = queryValue.execute(db); + if (response == null) { + throw new LSHException(db.getLastError().message); + } + val = (Float) response.resultArray[0]; + if (val == null) { + throw new LSHException("Self-score not recorded for " + md5); + } + cacheMap.put(md5, val); + return val.floatValue(); + } + + @Override + public void commitSelfScore(String md5, float score) throws LSHException { + initialize(); + Float val = score; + cacheMap.put(md5, val); + setUpInsert(1); + insertValue.keys[0] = md5; + insertValue.values[0] = val; + ResponseOptionalExist response = insertValue.execute(db); + if (response == null) { + throw new LSHException(db.getLastError().message); + } + } + + @Override + public double getSimThreshold() throws LSHException { + initialize(); + return simThreshold; + } + + @Override + public double getSigThreshold() throws LSHException { + initialize(); + return sigThreshold; + } + + @Override + public void resetStorage(double simThresh, double sigThresh) throws LSHException { + simThreshold = simThresh; + sigThreshold = sigThresh; + cacheMap = new TreeMap(); // Clear the cache + QueryOptionalExist query = new QueryOptionalExist(); + query.tableName = TABLE_NAME; + query.keyType = Types.VARCHAR; + query.valueType = Types.REAL; + query.attemptCreation = false; + query.clearTable = true; // Clear out the table + ResponseOptionalExist response = query.execute(db); + if (response == null) { + throw new LSHException(db.getLastError().message); + } + if (!response.tableExists) { + throw new LSHException("Optional table does not exist when it should: " + TABLE_NAME); + } + setUpInsert(2); + insertValue.keys[0] = SIMILARITY_KEY; + insertValue.keys[1] = SIGNIFICANCE_KEY; + insertValue.values[0] = Float.valueOf((float) simThreshold); // Write thresholds to special table rows + insertValue.values[1] = Float.valueOf((float) sigThreshold); + if (insertValue.execute(db) == null) { + throw new LSHException("Unable to initialize new thresholds: " + TABLE_NAME); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/TemporaryScoreCaching.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/TemporaryScoreCaching.java new file mode 100755 index 0000000000..5881d85c4c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/TemporaryScoreCaching.java @@ -0,0 +1,95 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import java.util.*; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.ExecutableRecord; + +/** + * An in-memory score cacher. It supports commitSelfScore() and getSelfScore() + * calls, but the commits have no backing storage and vanish with each new instantiation + * of this object. + */ +public class TemporaryScoreCaching implements ScoreCaching { + + private TreeMap cacheMap; // In memory cache of scores, indexed by md5 string + private double simThreshold; // similarity threshold loaded from file + private double sigThreshold; // significance threshold loaded from file + + public TemporaryScoreCaching() { + cacheMap = null; + simThreshold = -1.0; // Negative indicates thresholds have not been configured + sigThreshold = -1.0; + } + + @Override + public void prefetchScores(Set exeSet, List missing) + throws LSHException { + if (missing != null) { + if (cacheMap == null) { + for (ExecutableRecord exeRec : exeSet) { + missing.add(exeRec); // Everything is missing + } + } + else { + for (ExecutableRecord exeRec : exeSet) { + if (!cacheMap.containsKey(exeRec.getMd5())) { + missing.add(exeRec); + } + } + } + } + } + + @Override + public float getSelfScore(String md5) throws LSHException { + if (cacheMap != null) { + Float val = cacheMap.get(md5); + if (val != null) { + return val.floatValue(); + } + } + throw new LSHException("Self-score not recorded for " + md5); + } + + @Override + public void commitSelfScore(String md5, float score) throws LSHException { + if (cacheMap == null) { + cacheMap = new TreeMap(); + } + cacheMap.put(md5, score); + } + + @Override + public double getSimThreshold() throws LSHException { + return simThreshold; + } + + @Override + public double getSigThreshold() throws LSHException { + return sigThreshold; + } + + @Override + public void resetStorage(double simThresh, double sigThresh) throws LSHException { + cacheMap = null; + simThreshold = simThresh; + sigThreshold = sigThresh; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/CachedStatement.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/CachedStatement.java new file mode 100644 index 0000000000..89000d89f5 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/CachedStatement.java @@ -0,0 +1,121 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.SQLException; +import java.sql.Statement; + +import ghidra.util.Msg; + +/** + * {@link CachedStatement} provides a cached {@link Statement} container which is intended to + * supply a reusable instance for use within a single thread. Attempts to use the statement + * in multiple threads is considered unsafe. + * + * @param {@link Statement} implementation class + */ +public class CachedStatement { + + private S statement; + private Thread ownerThread; + + /** + * Get the associated cached {@link Statement} or prepare one via the specified + * {@code statementSupplier} if not yet established. Tf the supplier is used + * the owner thread for the statement will be established based on the + * {@link Thread#currentThread()}. + * + * @param statementSupplier statement supplier function which must return a valid + * instance or throw an exception. + * @return statement + * @throws SQLException if supplier fails to produce a statement + * @throws RuntimeException if the current thread does not correspond to the owner + * thread of a previously established statement. This is considered a programming + * error if this occurs. + */ + public S prepareIfNeeded(StatementSupplier statementSupplier) throws SQLException { + S s = getStatement(); + if (s != null) { + return s; + } + s = statementSupplier.get(); + setStatement(s); + return s; + } + + /** + * Set the associated {@link Statement} instance. This method may be used in place of + * {@link #prepareIfNeeded(StatementSupplier)} although it is not preferred since it + * can result in replacement of one previously established. The {@link #getStatement()} + * should be used first to ensure one was not previously set. An error will be logged + * if the invocation replaces an existing statement which will be forced closed. + * + * The owner thread for the statement will be established based on the + * {@link Thread#currentThread()}. + * + * @param s statement to be cached + */ + public void setStatement(S s) { + S oldStatement = statement; + statement = s; + ownerThread = Thread.currentThread(); + if (oldStatement != null) { + Msg.error(this, "Statement cached more than once - closing old statement"); + try { + oldStatement.close(); + } + catch (SQLException e) { + // ignore + } + } + } + + /** + * Get the current cached {@link Statement}. + * + * @return the current cached {@link Statement} or null if not yet established. + * @throws RuntimeException if the current thread does not correspond to the owner + * thread of a previously established statement. This is considered a programming + * error if this occurs. + */ + public S getStatement() { + Thread t = Thread.currentThread(); + if (statement != null && !ownerThread.equals(t)) { + Msg.error(this, "BSim cached statement used in unsafe-thread manner:" + + "\n Created in: " + ownerThread.getName() + "\n Used in: " + t.getName()); + throw new RuntimeException("BSim cached statement used in unsafe-thread manner"); + } + return statement; + } + + /** + * Close the currently cached {@link Statement}. This method may be invoked + * from any thread but should be properly coordinated with its use in the statement + * owner thread. + */ + public void close() { + if (statement != null) { + try { + statement.close(); + } + catch (SQLException e) { + // ignore + } + statement = null; + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/CallgraphTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/CallgraphTable.java new file mode 100755 index 0000000000..64fa1251cb --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/CallgraphTable.java @@ -0,0 +1,133 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; +import java.util.ArrayList; +import java.util.List; + +import javax.help.UnsupportedOperationException; + +import ghidra.features.bsim.query.description.*; + +public class CallgraphTable extends SQLComplexTable { + + private static final String SELECT_STMT = "SELECT ALL * FROM callgraphtable WHERE src = ?"; + private static final String INSERT_STMT = "INSERT INTO callgraphtable (src,dest) VALUES(?,?)"; + + public static class CallgraphRow { + public long src; + public long dest; + } + + private final CachedStatement selectStatement = new CachedStatement<>(); + private final CachedStatement insertStatement = new CachedStatement<>(); + + public CallgraphTable() { + super("callgraphtable", "src"); + } + + @Override + public void close() { + selectStatement.close(); + insertStatement.close(); + super.close(); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate( + "CREATE TABLE callgraphtable (src BIGINT,dest BIGINT,PRIMARY KEY (src,dest) )"); + } + + @Override + public void drop(Statement st) throws SQLException { + throw new UnsupportedOperationException("CallgraphTable may not be dropped"); + } + + /** + * + * @param pgres the result set to extract from + * @return the new {@link CallgraphRow} + * @throws SQLException if there's an error parsing the {@link ResultSet} + */ + public static CallgraphRow extractCallgraphRow(ResultSet pgres) throws SQLException { + CallgraphRow res = new CallgraphRow(); + res.src = pgres.getLong(1); + res.dest = pgres.getLong(2); + return res; + } + + /** + * + * @param func the function description + * @param trackcallgraph true if the database tracks call graph information + * @return the list of {@link CallgraphRow}s + * @throws SQLException if there is a problem parsing the {@link ResultSet} objects + */ + public List queryCallgraphRows(FunctionDescription func, + boolean trackcallgraph) throws SQLException { + if (!trackcallgraph) { + throw new SQLException("SQL database does not have callgraph information enabled"); + } + PreparedStatement s = + selectStatement.prepareIfNeeded(() -> db.prepareStatement(SELECT_STMT)); + RowKey funcid = func.getId(); + if (funcid == null) { + throw new SQLException("FunctionDescription does not have id"); + } + s.setLong(1, funcid.getLong()); + try (ResultSet rs = s.executeQuery()) { + List callvec = new ArrayList<>(); + while (rs.next()) { + CallgraphRow row = extractCallgraphRow(rs); + if (row.dest == 0) { + continue; + } + callvec.add(row); + } + return callvec; + } + } + + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 1 || + !(arguments[0] instanceof FunctionDescription)) { + throw new IllegalArgumentException( + "Insert method for CallgraphTable must take exactly one FunctionDescription argument"); + } + + FunctionDescription func = (FunctionDescription) arguments[0]; + + PreparedStatement s = + insertStatement.prepareIfNeeded(() -> db.prepareStatement(INSERT_STMT)); + long srcid = func.getId().getLong(); + List callvec = func.getCallgraphRecord(); + if (callvec != null) { + for (CallgraphEntry element : callvec) { + FunctionDescription destFunc = element.getFunctionDescription(); + long destid = destFunc.getId().getLong(); + s.setLong(1, srcid); + s.setLong(2, destid); + s.executeUpdate(); + } + } + + return 0; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/DescriptionTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/DescriptionTable.java new file mode 100755 index 0000000000..f9de7621b5 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/DescriptionTable.java @@ -0,0 +1,416 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; +import java.util.ArrayList; +import java.util.List; + +import javax.help.UnsupportedOperationException; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.client.RowKeySQL; +import ghidra.features.bsim.query.client.tables.ExeTable.ExecutableRow; +import ghidra.features.bsim.query.description.*; + +/** + * This is the SQL table "desctable", which holds one row for each function ingested into the database. + * A row (DescriptionRow) consists of basic meta-data about the function: name, address, executable + */ +public class DescriptionTable extends SQLComplexTable { + + //@formatter:off + private static final String INSERT_STMT = + "INSERT INTO desctable (id,name_func,id_exe,id_signature,flags,addr) VALUES(DEFAULT,?,?,?,?,?)"; + private static final String SELECT_BY_SIGNATURE_ID_STMT = + "SELECT ALL * FROM desctable WHERE id_signature = "; // used as simple Statement w/ appended param + private static final String SELECT_BY_FUNC_NAMEADDR_STMT = + "SELECT ALL * FROM desctable WHERE name_func = ? AND addr = ? AND id_exe = ?"; + private static final String SELECT_BY_FUNC_NAME_STMT = + "SELECT ALL * FROM desctable WHERE name_func = ? AND id_exe = ?"; + //@formatter:on + + public static class DescriptionRow { + public long rowid; // Row id of the function within -desctable- + public String func_name; // Name of the function + public long id_exe; // Row id of the executable (within exetable) containing function + public long id_sig; // Row id of the feature vector (within vectortable) describing function + public long addr; // The (starting) address of the function + public int flags; // bit vector describing tags that are active for this function + } + + private String selectWithFilterTableClause = null; + private String selectWithFilterWhereClause = null; + private final CachedStatement selectWithFilterStatement = + new CachedStatement<>(); + + private final CachedStatement selectByFuncAddrAndExeStatement = + new CachedStatement<>(); + private final CachedStatement selectByFuncAndExeStatement = + new CachedStatement<>(); + private final CachedStatement insertStatement = new CachedStatement<>(); + private final CachedStatement selectByRowIdStatement = + new CachedStatement<>(); + + private ExeTable exeTable = null; + + public DescriptionTable(ExeTable exeTable) { + super("desctable", "id_exe"); + this.exeTable = exeTable; + } + + @Override + public void close() { + selectWithFilterStatement.close(); + selectByFuncAddrAndExeStatement.close(); + selectByFuncAndExeStatement.close(); + insertStatement.close(); + selectByRowIdStatement.close(); + super.close(); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate("CREATE TABLE desctable" + + " (id BIGSERIAL PRIMARY KEY,name_func TEXT,id_exe INTEGER,id_signature BIGINT,flags INTEGER,addr BIGINT)"); + st.executeUpdate("CREATE INDEX sigindex ON desctable (id_signature)"); + st.executeUpdate("CREATE INDEX exefuncindex ON desctable (id_exe,name_func,addr)"); + } + + @Override + public void drop(Statement st) throws SQLException { + throw new UnsupportedOperationException("DescriptionTable may not be dropped"); + } + + /** + * Given the row id of the function within desctable, extract the FunctionDescription object + * @param descManager is the container which will hold the object + * @param rowId is the row id of the function within desctable + * @return the FunctionDescription + * @throws SQLException if there is a problem creating or executing the query + * @throws LSHException if there is a problem parsing the result set + */ + public FunctionDescription querySingleDescriptionId(DescriptionManager descManager, long rowId) + throws SQLException, LSHException { + PreparedStatement s = selectByRowIdStatement.prepareIfNeeded( + () -> db.prepareStatement("SELECT ALL * FROM desctable WHERE id = ?")); + s.setLong(1, rowId); + try (ResultSet rs = s.executeQuery()) { + FunctionDescription fres; + fres = extractSingleDescriptionRow(rs, descManager); + if (fres == null) { + throw new SQLException("No desctable rows matching id"); + } + return fres; + } + } + + /** + * Process a result set (expected to contain a single row) from desctable, + * building the FunctionDescription object described by the row + * @param resultSet is the result set + * @param descManager is the container that will hold the resulting FunctionDescription + * @return the new FunctionDescription + * @throws SQLException if there is a problem creating or executing the query + * @throws LSHException if there is a problem creating the executable record + */ + private FunctionDescription extractSingleDescriptionRow(ResultSet resultSet, + DescriptionManager descManager) throws SQLException, LSHException { + boolean finished = false; + DescriptionRow currow = null; + + while (resultSet.next()) { + if (!finished) { + currow = new DescriptionRow(); + extractDescriptionRow(resultSet, currow); + finished = true; + } + } + if (currow == null) { + return null; + } + + RowKey rowKey = new RowKeySQL(currow.id_exe); + ExecutableRecord curexe = descManager.findExecutableByRow(rowKey); + if (curexe == null) { + ExecutableRow exerow = exeTable.querySingleExecutableId(currow.id_exe); + curexe = exeTable.makeExecutableRecord(descManager, exerow); + descManager.cacheExecutableByRow(curexe, rowKey); + } + + return convertDescriptionRow(currow, curexe, descManager, null); + } + + /** + * Given rows from desctable describing functions of a single executable, + * build the list of corresponding FunctionDescription objects + * @param descList is resulting list of FunctionDescriptions + * @param rowList is the list of DescriptionRows + * @param executable is the ExecutableRecord of the single executable + * @param descManager is the container to hold the new FunctionDescriptions + * @param sigRecord is a single SignatureRecord to associate with any new + * FunctionDescription (can be null) + */ + public void convertDescriptionRows(List descList, + List rowList, ExecutableRecord executable, + DescriptionManager descManager, SignatureRecord sigRecord) { + if (rowList.isEmpty()) { + return; + } + DescriptionRow currow = rowList.get(0); + + FunctionDescription fres = + convertDescriptionRow(currow, executable, descManager, sigRecord); + descList.add(fres); + if (sigRecord != null) { + descManager.setSignatureId(sigRecord, currow.id_sig); + } + for (int i = 1; i < rowList.size(); ++i) { + currow = rowList.get(i); + fres = convertDescriptionRow(currow, executable, descManager, sigRecord); + descList.add(fres); + } + } + + /** + * Given a function's raw meta-data from a desctable row (DescriptionRow), build the + * corresponding high-level FunctionDescription + * @param descRow is the function's row meta-data + * @param exeRecord is the ExecutableRecord for executable containing the function + * @param descManager is the container that will hold the new FunctionDescription + * @param sigRecord is SignatureRecord associated with the function (may be null) + * @return the new FunctionDescription + */ + public static FunctionDescription convertDescriptionRow(DescriptionRow descRow, + ExecutableRecord exeRecord, DescriptionManager descManager, SignatureRecord sigRecord) { + FunctionDescription fres = + descManager.newFunctionDescription(descRow.func_name, descRow.addr, exeRecord); + descManager.setFunctionDescriptionId(fres, new RowKeySQL(descRow.rowid)); + descManager.setFunctionDescriptionFlags(fres, descRow.flags); + descManager.setSignatureId(fres, descRow.id_sig); + if (sigRecord != null) { + descManager.attachSignature(fres, sigRecord); + } + return fres; + } + + /** + * Extract column meta-data of a desctable row from the SQL result set + * @param resultSet is the low-level result set (returned by an SQL query) + * @param descRow is the DescriptionRow + * @throws SQLException if there is a problem parsing the result set + */ + public static void extractDescriptionRow(ResultSet resultSet, DescriptionRow descRow) + throws SQLException { + descRow.rowid = resultSet.getLong(1); + descRow.func_name = resultSet.getString(2); + descRow.id_exe = resultSet.getInt(3); + descRow.id_sig = resultSet.getLong(4); + descRow.flags = resultSet.getInt(5); + descRow.addr = resultSet.getLong(6); + } + + /** + * Extract a list of desctable rows from the SQL result set + * Only build up to -max- DescriptionRow objects, but still run through all rows in the set. + * @param resultSet is the ResultSet to run through + * @param maxRows is the maximum number of DescriptionRows to build + * @return a list of the new DescriptionRows + * @throws SQLException if there is a problem parsing the result set + */ + public List extractDescriptionRows(ResultSet resultSet, int maxRows) + throws SQLException { + List descvec = new ArrayList<>(); + boolean finished = false; + + while (resultSet.next()) { + if (!finished) { + DescriptionRow row = new DescriptionRow(); + descvec.add(row); + extractDescriptionRow(resultSet, row); + if ((maxRows > 0) && (descvec.size() >= maxRows)) { + // maximum number of + // rows + finished = true; + } + } + } + return descvec; + } + + /** + * Assuming all the necessary ids have been filled in, store the function as a row in desctable + * + * @param arguments must be a single {@link FunctionDescription} + * @throws SQLException if there is a problem creating or executing the query + */ + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 1 || + !(arguments[0] instanceof FunctionDescription)) { + throw new IllegalArgumentException( + "Insert method for KeyValueTable must take exactly one FunctionDescription argument"); + } + + FunctionDescription func = (FunctionDescription) arguments[0]; + SignatureRecord srec = func.getSignatureRecord(); + long vecid = 0; + if (srec != null) { + vecid = srec.getVectorId(); + } + + PreparedStatement s = insertStatement.prepareIfNeeded( + () -> db.prepareStatement(INSERT_STMT, Statement.RETURN_GENERATED_KEYS)); + s.setString(1, func.getFunctionName()); + s.setInt(2, (int) func.getExecutableRecord().getRowId().getLong()); + s.setLong(3, vecid); + s.setInt(4, func.getFlags()); + s.setLong(5, func.getAddress()); + s.executeUpdate(); + + try (ResultSet rs = s.getGeneratedKeys()) { + if (!rs.next()) { + throw new SQLException("Did not get desctable sequence number after insertion"); + } + return rs.getLong(1); + } + } + + /** + * Return function DescriptionRow objects that have a matching vector id + * + * @param vectorId is the row id of the feature vector we want to match + * @param maxRows is the maximum number of function rows to return + * @return list of resulting DescriptionRows + * @throws SQLException if there is a problem creating or executing the query + */ + public List queryVectorIdMatch(long vectorId, int maxRows) + throws SQLException { + + StringBuffer buf = new StringBuffer(); + buf.append(SELECT_BY_SIGNATURE_ID_STMT).append(vectorId); + if (maxRows > 0) { + buf.append(" LIMIT ").append(maxRows); + } + + try (Statement select_desctable_id_signature_stmt = db.createStatement(); + ResultSet rs = select_desctable_id_signature_stmt.executeQuery(buf.toString())) { + select_desctable_id_signature_stmt.setFetchSize(50); + List descres = null; + descres = extractDescriptionRows(rs, maxRows); + return descres; + } + } + + /** + * Return function DescriptionRow objects that have a matching vector id + * and that also pass additional filters. The filters must be encoded + * as a "WHERE" clause of an SQL "SELECT" statement on desctable. Additional + * tables joined to desctable to satisfy the filter must be encoded as + * a "FROM" clause of the "SELECT". + * @param vectorId is the row id of the feature vector (vectortable) we want to match on + * @param tableClause is the additional "FROM" clause needed for the filter + * @param whereClause is the "WHERE" clause needed for the filter + * @param maxRows is the maximum number of rows to return + * @return a list of resulting DescriptionRows + * @throws SQLException if there is an error creating or executing the query + */ + public List queryVectorIdMatchFilter(long vectorId, String tableClause, + String whereClause, int maxRows) throws SQLException { + + PreparedStatement s = selectWithFilterStatement.getStatement(); + if (s == null || !tableClause.equals(selectWithFilterTableClause) || + !whereClause.equals(selectWithFilterWhereClause)) { + selectWithFilterStatement.close(); + selectWithFilterTableClause = tableClause; + selectWithFilterWhereClause = whereClause; + StringBuffer buf = new StringBuffer(); + buf.append( + "SELECT desctable.id,desctable.name_func,desctable.id_exe,desctable.id_signature,desctable.flags,desctable.addr FROM desctable"); + buf.append(tableClause); + buf.append(" WHERE desctable.id_signature = ? "); + buf.append(whereClause); + buf.append(" LIMIT ?"); + s = db.prepareStatement(buf.toString()); + selectWithFilterStatement.setStatement(s); + } + + if (maxRows == 0) { + maxRows = 1000000; + } + s.setLong(1, vectorId); + s.setInt(2, maxRows); + s.setFetchSize(50); + try (ResultSet rs = s.executeQuery()) { + return extractDescriptionRows(rs, maxRows); + } + } + + /** + * Return DescriptionRow objects that match a given -functionName- + * and the row id within exetable of a specific executable + * @param executableId is the row id of the executable to match + * @param functionName is the name of the function to match + * @param maxRows is the maximum number of functions to return + * @return linked of DescriptionRow objects + * @throws SQLException if there is an error creating or executing the query + */ + public List queryFuncName(long executableId, String functionName, + int maxRows) throws SQLException { + PreparedStatement s = selectByFuncAndExeStatement + .prepareIfNeeded(() -> db.prepareStatement(SELECT_BY_FUNC_NAME_STMT)); + s.setString(1, functionName); + s.setInt(2, (int) executableId); + s.setFetchSize(50); + try (ResultSet rs = s.executeQuery()) { + return extractDescriptionRows(rs, maxRows); + } + } + + /** + * Query the description table for the row describing a single function. + * A function is uniquely identified by: its name, address, and the executable it is in + * @param executableId is the row id (of exetable) of the executable containing the function + * @param functionName is the name of the function + * @param functionAddress is the address of the function + * @return the corresponding row of the table, or null + * @throws SQLException if there is an error creating or executing the query + */ + public DescriptionRow queryFuncNameAddr(long executableId, String functionName, + long functionAddress) throws SQLException { + PreparedStatement s = selectByFuncAddrAndExeStatement + .prepareIfNeeded(() -> db.prepareStatement(SELECT_BY_FUNC_NAMEADDR_STMT)); + s.setString(1, functionName); + s.setLong(2, functionAddress); + s.setInt(3, (int) executableId); + s.setFetchSize(3); // Should only every be at most 1 + + DescriptionRow resultRow = null; + try (ResultSet rs = s.executeQuery()) { + boolean finished = false; + // TODO: Should be no need to get every row + while (rs.next()) { // To ensure JDBC resources are cleaned up make sure to visit every element + if (!finished) { // of the result set even if we don't build a DescriptionRow for every element + resultRow = new DescriptionRow(); + extractDescriptionRow(rs, resultRow); + finished = true; + } + } + return resultRow; + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/ExeTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/ExeTable.java new file mode 100755 index 0000000000..466d672018 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/ExeTable.java @@ -0,0 +1,700 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; +import java.text.SimpleDateFormat; +import java.time.*; +import java.util.ArrayList; +import java.util.Date; +import java.util.List; + +import org.apache.commons.lang3.StringUtils; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.client.AbstractSQLFunctionDatabase; +import ghidra.features.bsim.query.client.RowKeySQL; +import ghidra.features.bsim.query.description.*; + +public class ExeTable extends SQLComplexTable { + + public static final String TABLE_NAME = "exetable"; + + //@formatter:off + private static final String CREATE_STMT = + "CREATE TABLE " + TABLE_NAME + + " (id SERIAL PRIMARY KEY,md5 TEXT UNIQUE,name_exec TEXT,architecture INTEGER," + + " name_compiler INTEGER,ingest_date TIMESTAMP WITH TIME ZONE,repository INTEGER," + + " path INTEGER)"; + + private static final String INSERT_STMT = + "INSERT INTO " + TABLE_NAME + + " (id,md5,name_exec,architecture,name_compiler,ingest_date,repository,path)" + + " VALUES(DEFAULT,?,?,?,?,?,?,?) "; + + private static final String SELECT_BY_NAME_STMT = + "SELECT ALL id,md5,name_exec,architecture,name_compiler,extract(epoch from ingest_date),repository,path" + + " FROM " + TABLE_NAME + + " WHERE name_exec = ? " + + " LIMIT ?"; + + private static final String SELECT_BY_ID_STMT = + "SELECT ALL id,md5,name_exec,architecture,name_compiler,extract(epoch from ingest_date),repository,path" + + " FROM " + TABLE_NAME + + " WHERE id = ?"; + + private static final String SELECT_BY_MD5_STMT = + "SELECT ALL id,md5,name_exec,architecture,name_compiler,extract(epoch from ingest_date),repository,path" + + " FROM " + TABLE_NAME + + " WHERE md5 = ?"; + + //@formatter:on + + private final SQLStringTable archtable; + private final SQLStringTable compilertable; + private final SQLStringTable repositorytable; + private final SQLStringTable pathtable; + + private final ExeToCategoryTable exeCategoryTable; + + public static enum ExeTableOrderColumn { + MD5, NAME + } + + public static class ExecutableRow { + public long rowid; + public String md5; + public String exename; + public long arch_id; + public long compiler_id; + public long date_milli; + public long repo_id; + public long path_id; + } + + private final CachedStatement insertRowStatement = new CachedStatement<>(); + private final CachedStatement selectByIdStatement = new CachedStatement<>(); + private final CachedStatement selectByMd5Statement = new CachedStatement<>(); + private final CachedStatement selectByNameStatement = + new CachedStatement<>(); + + /** + * Constructor + * + * @param archtable the architecture table + * @param compilertable the compiler table + * @param repositorytable the repository table + * @param pathtable the path table + * @param exeCategoryTable the category table + */ + public ExeTable(SQLStringTable archtable, SQLStringTable compilertable, + SQLStringTable repositorytable, SQLStringTable pathtable, + ExeToCategoryTable exeCategoryTable) { + super(TABLE_NAME, "id"); + this.archtable = archtable; + this.compilertable = compilertable; + this.repositorytable = repositorytable; + this.pathtable = pathtable; + this.exeCategoryTable = exeCategoryTable; + } + + @Override + public void close() { + insertRowStatement.close(); + selectByIdStatement.close(); + selectByMd5Statement.close(); + selectByNameStatement.close(); + super.close(); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate(CREATE_STMT); + } + + @Override + public void drop(Statement st) throws SQLException { + throw new UnsupportedOperationException("ExeTable may not be dropped"); + } + + @Override + public int delete(long id) throws SQLException { + int rowcount = super.delete(id); + if (rowcount == 0) { + throw new SQLException("Could not delete executable record"); + } + if (rowcount > 1) { + throw new SQLException("Problem deleting executable record"); + } + return rowcount; + } + + /** + * Pulls information out of the given {@link ExecutableRow} object into the given + * {@link ResultSet} + * + * @param pgres the result set + * @param res the executable row + * @throws SQLException if there is a problem parsing the result set + */ + protected static void extractExecutableRow(ResultSet pgres, ExecutableRow res) + throws SQLException { + res.rowid = pgres.getInt(1); + res.md5 = pgres.getString(2); + res.exename = pgres.getString(3); + res.arch_id = pgres.getInt(4); + res.compiler_id = pgres.getInt(5); + res.date_milli = (long) (pgres.getDouble(6) * 1000.0); + res.repo_id = pgres.getInt(7); + res.path_id = pgres.getInt(8); + } + + /** + * Creates {@link ExecutableRecord} objects from {@link ResultSet} and stores + * them in the given list. + * + * @param rs the result set + * @param vecres the list of executable records + * @param res the description manager + * @param max the max number of rows to return + * @return the number of rows returned + * @throws SQLException if there is an problem parsing the result set + * @throws LSHException if there is an problem creating the executable record + */ + public int extractExecutableRows(ResultSet rs, List vecres, + DescriptionManager res, int max) throws SQLException, LSHException { + + // Assume the sql statement is returning some number of exetable rows + // Read all these rows and create corresponding ExecutableRecord objects + List exerows = new ArrayList<>(); + boolean finished = false; + + while (rs.next()) { + if (!finished) { + ExecutableRow row = new ExecutableRow(); + exerows.add(row); + extractExecutableRow(rs, row); + if ((max > 0) && (exerows.size() >= max)) { + finished = true; + } + } + } + + for (ExecutableRow exerow : exerows) { + ExecutableRecord exerec = makeExecutableRecord(res, exerow); + if (vecres != null) { + vecres.add(exerec); + } + } + return exerows.size(); + } + + /** + * Make an ExecutableRecord within the DescriptionManager container, given + * database row information + * + * @param manager is the DescriptionManager that will contain the new record + * @param row is the columnar values for the executable from the database + * @return the new ExecutableRecord + * @throws SQLException if there is a problem parsing the table objects + * @throws LSHException if there is a problem creating a new exec library or record + */ + public ExecutableRecord makeExecutableRecord(DescriptionManager manager, ExecutableRow row) + throws SQLException, LSHException { + String arch = archtable.getString(row.arch_id); + + ExecutableRecord exerec; + RowKeySQL rowid = new RowKeySQL(row.rowid); + if (ExecutableRecord.isLibraryHash(row.md5)) { + exerec = manager.newExecutableLibrary(row.exename, arch, rowid); + } + else { + String cname = compilertable.getString(row.compiler_id); + String repo = repositorytable.getString(row.repo_id); + String path = pathtable.getString(row.path_id); + Date date = new Date(row.date_milli); + exerec = manager.newExecutableRecord(row.md5, row.exename, cname, arch, date, repo, + path, rowid); + } + return exerec; + } + + /** + * Query for a unique executable based on -name- and possibly other metadata + * + * @param manage the container to store the result + * @param name the name the executable must match + * @param arch the architecture the executable must match (may be zero length) + * @param cname the compiler name the executable must match (may be zero length) + * @return the unique resulting ExecutableRecord or null, if none or more + * than 1 is found + * @throws SQLException if there is a problem querying for the executable name + * @throws LSHException if there is a problem querying for the executable name or transferring the exec + */ + public ExecutableRecord querySingleExecutable(DescriptionManager manage, String name, + String arch, String cname) throws SQLException, LSHException { + DescriptionManager tmp = new DescriptionManager(); + queryNameExeMatch(null, tmp, name, 2000000); + ExecutableRecord res = null; + int count = 0; + + for (ExecutableRecord erec : tmp.getExecutableRecordSet()) { + if (!StringUtils.isBlank(arch)) { + if (!erec.getArchitecture().equals(arch)) { + continue; + } + } + if (!StringUtils.isBlank(cname)) { + if (!erec.getNameCompiler().equals(cname)) { + continue; + } + } + res = erec; + count += 1; + if (count > 1) { + return null; // More than one match + } + } + if (count != 1) { + return null; // No matches + } + return manage.transferExecutable(res); + } + + /** + * Executes a database query to return a list of records matching an executalble name + * filter. + * + * @param vecres the list of executable records to populate + * @param res the description manager + * @param nm the name to query for + * @param max the max number of records to return + * @return the number of records returned + * @throws SQLException if there is a problem creating the query statement + * @throws LSHException if there is a problem extracting executable rows + */ + public int queryNameExeMatch(List vecres, DescriptionManager res, + String nm, int max) throws SQLException, LSHException { + PreparedStatement s = + selectByNameStatement.prepareIfNeeded(() -> db.prepareStatement(SELECT_BY_NAME_STMT)); + s.setString(1, nm); + s.setInt(2, max > 0 ? max : 2000000); + s.setFetchSize(50); + int count = 0; + try (ResultSet rs = s.executeQuery()) { + count = extractExecutableRows(rs, vecres, res, max); + } + return count; + } + + /** + * Query for a single executable based on its exetable -id- + * + * @param id the exetable id + * @return the executable row + * @throws SQLException if there is a problem creating or executing the query + */ + public ExecutableRow querySingleExecutableId(long id) throws SQLException { + + PreparedStatement s = + selectByIdStatement.prepareIfNeeded(() -> db.prepareStatement(SELECT_BY_ID_STMT)); + s.setInt(1, (int) id); + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { + throw new SQLException("Bad exetable rowid"); + } + ExecutableRow row = new ExecutableRow(); + extractExecutableRow(rs, row); + return row; + } + } + + /** + * Return the executable with matching md5 (if any) + * + * @param md5 the md5 hash to query + * @return the ExecutableRow data or null + * @throws SQLException if there is a problem creating or executing the query + */ + public ExecutableRow queryMd5ExeMatch(String md5) throws SQLException { + + PreparedStatement s = + selectByMd5Statement.prepareIfNeeded(() -> db.prepareStatement(SELECT_BY_MD5_STMT)); + s.setString(1, md5); + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { // If there are no matching rows + return null; + } + ExecutableRow row = new ExecutableRow(); + extractExecutableRow(rs, row); + return row; + } + } + + /** + * Returns a count of all records in the database matching the filter criteria. + * + * @param filterMd5 md5 must contain this + * @param filterExeName exe name must contain this + * @param filterArch if non-zero, force matching architecture id + * @param filterCompilerName if non-zero, force matching compiler id + * @param includeFakes if true, include MD5s that start with 'bbbbbbbbaaaaaaa' + * @return total number of records in the database + * @throws SQLException when preparing or executing the query + */ + public int queryExeCount(String filterMd5, String filterExeName, long filterArch, + long filterCompilerName, boolean includeFakes) throws SQLException { + + if (StringUtils.isAnyBlank(filterMd5)) { + filterMd5 = null; + } + if (StringUtils.isAnyBlank(filterExeName)) { + filterExeName = null; + } + + StringBuilder buffer = new StringBuilder(); + buffer.append("SELECT ALL COUNT(*) FROM "); + buffer.append(TABLE_NAME); + buffer.append(" "); + if (filterMd5 != null || filterExeName != null || filterArch != 0 || + filterCompilerName != 0 || !includeFakes) { + buffer.append("WHERE ("); + boolean needAnd = false; + if (!includeFakes) { + buffer.append("md5 NOT ILIKE 'bbbbbbbbaaaaaaaa%' "); + needAnd = true; + } + if (filterMd5 != null) { + if (needAnd) { + buffer.append("AND "); + } + if (filterMd5.length() == 32) { // A complete md5 value + buffer.append("md5 = ? "); + } + else { // A partial md5 prefix + buffer.append("md5 ILIKE ? "); + filterMd5 = filterMd5 + '%'; // match anything at the end + } + needAnd = true; + } + if (filterExeName != null) { + if (needAnd) { + buffer.append("AND "); + } + buffer.append("name_exec ILIKE ? "); + filterExeName = '%' + filterExeName + '%'; + needAnd = true; + } + if (filterArch != 0) { + if (needAnd) { + buffer.append("AND "); + } + buffer.append("architecture = ? "); + needAnd = true; + } + if (filterCompilerName != 0) { + if (needAnd) { + buffer.append("AND "); + } + buffer.append("name_compiler = ? "); + } + buffer.append(')'); + } + + int pos = 1; + String query = buffer.toString(); + try (PreparedStatement selectExeCountStatement = db.prepareStatement(query)) { + if (filterMd5 != null) { + selectExeCountStatement.setString(pos++, filterMd5); + } + if (filterExeName != null) { + selectExeCountStatement.setString(pos++, filterExeName); + } + if (filterArch != 0) { + selectExeCountStatement.setInt(pos++, (int) filterArch); + } + if (filterCompilerName != 0) { + selectExeCountStatement.setInt(pos++, (int) filterCompilerName); + } + + try (ResultSet rs = selectExeCountStatement.executeQuery()) { + rs.next(); + int count = rs.getInt(1); + return count; + } + catch (Exception e) { + return 0; + } + } + } + + /** + * Returns a list of all rows in the exe table matching a given filter. + * + * @param limit the max number of results to return + * @param filterMd5 md5 must contain this + * @param filterExeName exe name must contain this + * @param filterArch if non-zero architecture must match this id + * @param filterCompilerName if non-zero compiler must match this id + * @param sortColumn the name of the column that should define the sorting order + * @param includeFakes if false, will exclude generated MD5s starting with "bbbbbbbbaaaaaaaa" + * @return list of executables + * @throws SQLException when preparing or executing the query + */ + public List queryAllExe(int limit, String filterMd5, String filterExeName, + long filterArch, long filterCompilerName, ExeTableOrderColumn sortColumn, + boolean includeFakes) throws SQLException { + + if (StringUtils.isAnyBlank(filterMd5)) { + filterMd5 = null; + } + if (StringUtils.isAnyBlank(filterExeName)) { + filterExeName = null; + } + StringBuilder buffer = new StringBuilder(); + buffer.append( + "SELECT ALL id,md5,name_exec,architecture,name_compiler,extract(epoch from ingest_date),repository,path FROM "); + buffer.append(TABLE_NAME); + buffer.append(" "); + if (filterMd5 != null || filterExeName != null || filterArch != 0 || + filterCompilerName != 0 || !includeFakes) { + buffer.append("WHERE ("); + boolean needAnd = false; + if (!includeFakes) { + buffer.append("md5 NOT ILIKE 'bbbbbbbbaaaaaaaa%' "); + needAnd = true; + } + if (filterMd5 != null) { + if (needAnd) { + buffer.append("AND "); + } + if (filterMd5.length() == 32) { // A complete md5 value + buffer.append("md5 = ? "); + } + else { // A partial md5 prefix + buffer.append("md5 ILIKE ? "); + filterMd5 = filterMd5 + '%'; // match anything at the end + } + needAnd = true; + } + if (filterExeName != null) { + if (needAnd) { + buffer.append("AND "); + } + buffer.append("name_exec ILIKE ? "); + filterExeName = '%' + filterExeName + '%'; + needAnd = true; + } + if (filterArch != 0) { + if (needAnd) { + buffer.append("AND "); + } + buffer.append("architecture = ? "); + needAnd = true; + } + if (filterCompilerName != 0) { + if (needAnd) { + buffer.append("AND "); + } + buffer.append("name_compiler = ? "); + } + buffer.append(')'); + } + if (sortColumn == ExeTableOrderColumn.NAME) { + buffer.append(" ORDER BY name_exec "); + } + else { + buffer.append(" ORDER BY md5 "); + } + buffer.append("LIMIT ?"); + + int pos = 1; + String query = buffer.toString(); + try (PreparedStatement selectAllExeStatement = db.prepareStatement(query)) { + if (filterMd5 != null) { + selectAllExeStatement.setString(pos++, filterMd5); + } + if (filterExeName != null) { + selectAllExeStatement.setString(pos++, filterExeName); + } + if (filterArch != 0) { + selectAllExeStatement.setInt(pos++, (int) filterArch); + } + if (filterCompilerName != 0) { + selectAllExeStatement.setInt(pos++, (int) filterCompilerName); + } + selectAllExeStatement.setInt(pos, limit); + + try (ResultSet rs = selectAllExeStatement.executeQuery()) { + List executables = new ArrayList<>(); + while (rs.next()) { + ExecutableRow row = new ExecutableRow(); + executables.add(row); + extractExecutableRow(rs, row); + } + + return executables; + } + } + } + + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 1 || + !(arguments[0] instanceof ExecutableRecord)) { + throw new IllegalArgumentException( + "Insert method for ExeTable must take exactly one ExecutableRecord argument"); + } + + ExecutableRecord erec = (ExecutableRecord) arguments[0]; + PreparedStatement s = insertRowStatement.prepareIfNeeded( + () -> db.prepareStatement(INSERT_STMT, Statement.RETURN_GENERATED_KEYS)); + long arch_id = archtable.writeString(erec.getArchitecture()); + long compiler_id = compilertable.writeString(erec.getNameCompiler()); + long repo_id = repositorytable.writeString(erec.getRepository()); + long path_id = pathtable.writeString(erec.getPath()); + long milli = erec.getDate().getTime(); + + OffsetDateTime odt = + OffsetDateTime.ofInstant(Instant.ofEpochMilli(milli), ZoneId.systemDefault()); + + s.setString(1, erec.getMd5()); + s.setString(2, erec.getNameExec()); + s.setInt(3, (int) arch_id); + s.setInt(4, (int) compiler_id); + s.setObject(5, odt); + s.setInt(6, (int) repo_id); + s.setInt(7, (int) path_id); + + s.executeUpdate(); + try (ResultSet rs = s.getGeneratedKeys()) { + if (!rs.next()) { + throw new SQLException("Did not get exetable sequence number after insertion"); + } + long rowid = rs.getLong(1); + return rowid; + } + } + + /** + * Updates records in the database with information in the given {@link ExecutableRecord}. + * + * @param rec the executable record to update + * @throws SQLException if there is a problem creating or executing the query + */ + public void updateExecutable(ExecutableRecord.Update rec) throws SQLException { + if (rec.architecture || rec.date || rec.name_compiler || rec.name_exec || rec.path || + rec.repository) { + + try (Statement st = db.createStatement()) { + StringBuilder buf = new StringBuilder(); + boolean previous = false; + buf.append("UPDATE "); + buf.append(TABLE_NAME); + buf.append(" SET "); + if (rec.name_exec) { + if (previous) { + buf.append(','); + } + else { + previous = true; + } + buf.append("name_exec='"); + AbstractSQLFunctionDatabase.appendEscapedLiteral(buf, rec.update.getNameExec()); + buf.append('\''); + } + if (rec.architecture) { + long arch_id = archtable.writeString(rec.update.getArchitecture()); + if (previous) { + buf.append(','); + } + else { + previous = true; + } + buf.append("architecture="); + buf.append(arch_id); + } + if (rec.name_compiler) { + long comp_id = compilertable.writeString(rec.update.getNameCompiler()); + if (previous) { + buf.append(','); + } + else { + previous = true; + } + buf.append("name_compiler="); + buf.append(comp_id); + } + if (rec.date) { + if (previous) { + buf.append(','); + } + else { + previous = true; + } + String dateString = + new SimpleDateFormat(AbstractSQLFunctionDatabase.JAVA_TIME_FORMAT) + .format(rec.update.getDate()); + buf.append("ingest_date=to_timestamp('"); + // Using PostgreSQL compatible to_timestamp formatting options + buf.append(dateString) + .append("','" + AbstractSQLFunctionDatabase.SQL_TIME_FORMAT + "')"); + } + if (rec.repository) { + long repo_id = repositorytable.writeString(rec.update.getRepository()); + if (previous) { + buf.append(','); + } + else { + previous = true; + } + buf.append("repository="); + buf.append(repo_id); + } + if (rec.path) { + long path_id = pathtable.writeString(rec.update.getPath()); + if (previous) { + buf.append(','); + } + else { + previous = true; + } + buf.append("path="); + buf.append(path_id); + } + buf.append(" WHERE id = "); + buf.append(rec.update.getRowId().getLong()); + + st.executeUpdate(buf.toString()); + } + } + + if (rec.categories) { // Change must be made to categories + List insertlist = rec.catinsert; + if (insertlist == null) { // A null is a flag for full deletion, + // followed by full insertion + exeCategoryTable.delete(rec.update.getRowId().getLong()); + insertlist = rec.update.getAllCategories(); + } + if (insertlist != null) { + for (CategoryRecord element : insertlist) { + exeCategoryTable.insert(element, rec.update.getRowId().getLong()); + } + } + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/ExeToCategoryTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/ExeToCategoryTable.java new file mode 100755 index 0000000000..7e73a58bdc --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/ExeToCategoryTable.java @@ -0,0 +1,173 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; +import java.util.ArrayList; +import java.util.List; + +import ghidra.features.bsim.query.description.CategoryRecord; +import ghidra.features.bsim.query.description.ExecutableRecord; + +public class ExeToCategoryTable extends SQLComplexTable { + + private static final String INSERT_STMT = + "INSERT INTO execattable (id_exe,id_type,id_category) VALUES(?,?,?)"; + private static final String SELECT_STMT = "SELECT ALL * FROM execattable WHERE id_exe = ?"; + + private final SQLStringTable catstringtable; + + private final CachedStatement selectCategoriesStatement = + new CachedStatement<>(); + private final CachedStatement insertExeCatStatement = + new CachedStatement<>(); + + protected static class CategoryRow { + public long id_exe; + public long id_type; + public long id_category; + } + + /** + * Constructor + * + * @param catstringtable table containing all category values + */ + public ExeToCategoryTable(SQLStringTable catstringtable) { + super("execattable", "id_exe"); + this.catstringtable = catstringtable; + } + + @Override + public void close() { + selectCategoriesStatement.close(); + insertExeCatStatement.close(); + super.close(); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate( + "CREATE TABLE execattable (id_exe INTEGER,id_type INTEGER,id_category INTEGER)"); + st.executeUpdate("CREATE INDEX execatindex ON execattable (id_exe,id_category)"); + } + + @Override + public void drop(Statement st) throws SQLException { + throw new UnsupportedOperationException("ExeToCategoryTable may not be dropped"); + } + + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 2 || + !(arguments[0] instanceof CategoryRecord)) { + throw new IllegalArgumentException( + "Insert method for ExeToCategoryTable must take exactly 2 arguments: CategoryRecord and a long(id_exe)"); + } + + CategoryRecord catrec = (CategoryRecord) arguments[0]; + long id_type = catstringtable.writeString(catrec.getType()); + long id_category = catstringtable.writeString(catrec.getCategory()); + long exe_id = (long) arguments[1]; + + // Insert a new row referencing that existing row into the execattable. + PreparedStatement s = + insertExeCatStatement.prepareIfNeeded(() -> db.prepareStatement(INSERT_STMT)); + s.setInt(1, (int) exe_id); + s.setInt(2, (int) id_type); + s.setInt(3, (int) id_category); + s.executeUpdate(); + + // Return value is meaningless here. + return 0; + + } + + protected static void extractCategoryRow(ResultSet pgres, CategoryRow res) throws SQLException { + res.id_exe = pgres.getInt(1); + res.id_type = pgres.getInt(2); + res.id_category = pgres.getInt(3); + } + + protected void extractCategoryRecords(ResultSet rs, List vecres, int max) + throws SQLException { + List catrows = new ArrayList(); + boolean finished = false; + + while (rs.next()) { + if (!finished) { + CategoryRow row = new CategoryRow(); + catrows.add(row); + extractCategoryRow(rs, row); + if ((max > 0) && (catrows.size() >= max)) { + finished = true; + } + } + } + for (int i = 0; i < catrows.size(); ++i) { + CategoryRow row = catrows.get(i); + String type = catstringtable.getString(row.id_type); + String category = catstringtable.getString(row.id_category); + CategoryRecord catrec = new CategoryRecord(type, category); + vecres.add(catrec); + } + } + + /** + * + * @param exeid the executable table id + * @param max the max number of records to return + * @return the list of category records + * @throws SQLException if there is a problem creating or executing the query + */ + public List queryExecutableCategories(long exeid, int max) + throws SQLException { + + if (exeid == 0) { + throw new SQLException("ExecutableRecord does not have id"); + } + + PreparedStatement s = + selectCategoriesStatement.prepareIfNeeded(() -> db.prepareStatement(SELECT_STMT)); + s.setInt(1, (int) exeid); + try (ResultSet rs = s.executeQuery()) { + List catvec = new ArrayList(); + extractCategoryRecords(rs, catvec, max); + return catvec; + } + } + + /** + * + * @param erec the executable record + * @throws SQLException if there is a problem inserting the category + */ + public void storeExecutableCategories(ExecutableRecord erec) throws SQLException { + if (erec.isAlreadyStored()) { + return; + } + // TODO: We are NOT checking if the stored exe has the same categories as this exe + List catrecs = erec.getAllCategories(); + if (catrecs == null) { + return; + } + long exeid = erec.getRowId().getLong(); + for (CategoryRecord catrec : catrecs) { + insert(catrec, exeid); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/IdfLookupTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/IdfLookupTable.java new file mode 100755 index 0000000000..d1da49e80c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/IdfLookupTable.java @@ -0,0 +1,92 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; + +import generic.lsh.vector.IDFLookup; + +public class IdfLookupTable extends SQLComplexTable { + + public IdfLookupTable() { + super("idflookup", null); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate("CREATE TABLE idflookup(hash bigint,lookup integer)"); + } + + @Override + public void drop(Statement st) throws SQLException { + String sql = "DROP TABLE IF EXISTS " + tableName; + st.executeUpdate(sql); + } + + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 2) { + throw new IllegalArgumentException( + "Insert method for IdfLookupTable must take exactly two integer arguments"); + } + + try (Statement st = db.createStatement()) { + + int cnt = (int) arguments[0]; + if (cnt == 0xffffffff) { + return 0; + } + StringBuffer buf = new StringBuffer(); + buf.append("INSERT INTO idflookup (hash,lookup) VALUES("); + long rawhash = (int) arguments[1]; + if (rawhash < 0) { + rawhash += 0x100000000L; + } + buf.append(rawhash); + buf.append(',').append(cnt).append(')'); + st.executeUpdate(buf.toString()); + + return 0; // return value is meaningless here + } + } + + /** + * + * @param lookup the IDF lookup + * @throws SQLException if there is an error creating/executing the query + */ + public void recoverIDFLookup(IDFLookup lookup) throws SQLException { + try (Statement st = db.createStatement(); + ResultSet rs = st.executeQuery("SELECT ALL * from idflookup")) { + int buffer[] = new int[5000]; + int numentries = 0; + while (rs.next()) { + buffer[numentries] = (int) rs.getLong(1); + numentries += 1; + buffer[numentries] = rs.getInt(2); + numentries += 1; + } + + int[] finalArray = new int[numentries]; + for (int i = 0; i < finalArray.length; ++i) { + finalArray[i] = buffer[i]; + } + lookup.set(finalArray); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/KeyValueTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/KeyValueTable.java new file mode 100755 index 0000000000..74cb2d7580 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/KeyValueTable.java @@ -0,0 +1,165 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; + +import ghidra.features.bsim.query.description.DatabaseInformation; + +public class KeyValueTable extends SQLComplexTable { + + private final String INSERT_STMT = "INSERT INTO keyvaluetable (key,value) VALUES(?,?)"; + private final String UPDATE_STMT = "UPDATE keyvaluetable SET value = ? WHERE key = ?"; + private final String SELECT_STMT = "SELECT value FROM keyvaluetable WHERE key = ?"; + + private final CachedStatement insertStatement = new CachedStatement<>(); + private final CachedStatement updateStatement = new CachedStatement<>(); + private final CachedStatement selectStatement = new CachedStatement<>(); + + public KeyValueTable() { + super("keyvaluetable", null); + } + + @Override + public void close() { + insertStatement.close(); + updateStatement.close(); + selectStatement.close(); + super.close(); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate("CREATE TABLE keyvaluetable (key TEXT UNIQUE,value TEXT)"); + } + + @Override + public void drop(Statement st) throws SQLException { + throw new UnsupportedOperationException("KeyValueTable may not be dropped"); + } + + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 2 || !(arguments[0] instanceof String) || + !(arguments[1] instanceof String)) { + throw new IllegalArgumentException( + "Insert method for KeyValueTable must take exactly two arguments: String and String"); + } + + String key = (String) arguments[0]; + String value = (String) arguments[1]; + + PreparedStatement s = + updateStatement.prepareIfNeeded(() -> db.prepareStatement(UPDATE_STMT)); + s.setString(2, key); + s.setString(1, value); + + if (s.executeUpdate() == 1) { + return 0; // Update was successful + } + + s = insertStatement.prepareIfNeeded(() -> db.prepareStatement(INSERT_STMT)); + s.setString(1, key); + s.setString(2, value); + s.executeUpdate(); + + return 0; + } + + /** + * Inserts some properties from the {@link DatabaseInformation} object to the table. + * + * @param info the database information + * @throws SQLException if the database info cannot be stored in the table + */ + public void writeBasicInfo(DatabaseInformation info) throws SQLException { + insert("name", info.databasename); + insert("owner", info.owner); + insert("description", info.description); + insert("major", Integer.toString(info.major)); + insert("minor", Integer.toString(info.minor)); + insert("settings", Integer.toString(info.settings)); + insert("layout", Integer.toString(info.layout_version)); + String tf = info.readonly ? "t" : "f"; + insert("readonly", tf); + tf = info.trackcallgraph ? "t" : "f"; + insert("trackcallgraph", tf); + String datename = info.dateColumnName; + if (datename == null) { + datename = "Ingest Date"; + } + insert("datecolumn", datename); + writeExecutableCategories(info); + writeFunctionTags(info); + } + + /** + * + * @param info the database information + * @throws SQLException if the table insert fails + */ + public void writeExecutableCategories(DatabaseInformation info) throws SQLException { + if (info.execats == null) { + insert("execatcount", "0"); + return; + } + insert("execatcount", Integer.toString(info.execats.size())); + for (int i = 0; i < info.execats.size(); ++i) { + String key = "execat" + Integer.toString(i + 1); + insert(key, info.execats.get(i)); + } + } + + /** + * @param info the database information + * @throws SQLException if the table insert fails + */ + public void writeFunctionTags(DatabaseInformation info) throws SQLException { + if (info.functionTags == null) { + insert("functiontagcount", "0"); + return; + } + insert("functiontagcount", Integer.toString(info.functionTags.size())); + for (int i = 0; i < info.functionTags.size(); ++i) { + String key = "functiontag" + Integer.toString(i + 1); + insert(key, info.functionTags.get(i)); + } + } + + /** + * + * @param key the key to get the value for + * @return the value associated with key or throw exception if key not present + * @throws SQLException if the sql statement cannot be parsed + */ + public String getValue(String key) throws SQLException { + PreparedStatement s = + selectStatement.prepareIfNeeded(() -> db.prepareStatement(SELECT_STMT)); + s.setString(1, key); + try (ResultSet rs = s.executeQuery()) { + String value; + if (rs.next()) { + value = rs.getString(1); + } + else { + throw new SQLException("Could not fetch key value: " + key); + } + return value; + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/OptionalTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/OptionalTable.java new file mode 100755 index 0000000000..41b57cd5e3 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/OptionalTable.java @@ -0,0 +1,244 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; + +/** + * Database table that has exactly two columns: key and value + * The column types are variable and are determined upon initialization. + * They are specified by giving an integer "type code" as listed in java.sql.Types + * The key column will be marked as UNIQUE. + * The JDBC driver will map between Java and SQL types. + * The readValue() writeValue() and deleteValue() methods simply take and return an Object. + */ +public class OptionalTable { + + private final String TABLE_EXISTS_STMT = "SELECT schemaname FROM pg_tables where tablename='#'"; + private final String GRANT_STMT = "GRANT SELECT ON # TO PUBLIC"; + private final String DELETE_ALL_STMT = "DELETE FROM #"; + + private final String INSERT_STMT = "INSERT INTO # (key,value) VALUES(?,?)"; + private final String UPDATE_STMT = "UPDATE # SET value = ? WHERE key = ?"; + private final String SELECT_STMT = "SELECT value FROM # WHERE key = ?"; + private final String DELETE_STMT = "DELETE FROM # WHERE key = ?"; + + private Connection db = null; // Connection to the database + + private final CachedStatement insertStatement = new CachedStatement<>(); + private final CachedStatement updateStatement = new CachedStatement<>(); + private final CachedStatement selectStatement = new CachedStatement<>(); + private final CachedStatement deleteStatement = new CachedStatement<>(); + private final CachedStatement reusableStatement = new CachedStatement<>(); + private final String lockString; + + private String name = null; // name of the table + private int keyType = -1; // Type of the key column + private int valueType = -1; // Type of the value column + + /** + * Construct this table for a specific connection + * @param nm is the formal SQL name of the table + * @param kType is the type-code of the key (as specified in java.sql.Types) + * @param vType is the type-code of the value (as specified in java.sql.Types) + * @param d is the connection to the SQL server + */ + public OptionalTable(String nm, int kType, int vType, Connection d) { + name = nm; + keyType = kType; + valueType = vType; + db = d; + lockString = "LOCK TABLE " + nm + " IN SHARE ROW EXCLUSIVE MODE"; + } + + private Statement getReusableStatement() throws SQLException { + return reusableStatement.prepareIfNeeded(() -> db.createStatement()); + } + + /** + * Lock the table for writing + * @throws SQLException if the server reports an error + */ + public void lockForWrite() throws SQLException { + getReusableStatement().execute(lockString); + } + + /** + * Given an sql type-code, return the sql type as a string (suitable for the CREATE TABLE command) + * @param type is the type-code + * @return the name of the type corresponding to the type-code + */ + private static String getSQLType(int type) { + switch (type) { + case Types.INTEGER: + return "INTEGER"; + case Types.VARCHAR: + return "TEXT"; + case Types.REAL: + return "REAL"; + } + return null; + } + + /** + * Replace '#' character in the -ptr- command with our -name- and return the string + * @param ptr is the string to be modified + * @return the modified string + */ + private String generateSQLCommand(String ptr) { + int first = ptr.indexOf('#'); + int second = ptr.indexOf('#', first + 1); + String res; + res = ptr.substring(0, first); + res += name; + if (second >= 0) { + res += ptr.substring(first + 1, second); + res += name; + res += ptr.substring(second + 1); + } + else { + res += ptr.substring(first + 1); + } + return res; + } + + /** + * @return the formal sql name of the table + */ + public String getName() { + return name; + } + + /** + * @return type-code of key column + */ + public int getKeyType() { + return keyType; + } + + /** + * @return type-code of value column + */ + public int getValueType() { + return valueType; + } + + /** + * Free any resources and relinquish references to the connection + */ + public void close() { + insertStatement.close(); + selectStatement.close(); + updateStatement.close(); + deleteStatement.close(); + reusableStatement.close(); + db = null; // We do not own the db + } + + /** + * Create this specific table in the database + * @throws SQLException for problems with the connection + */ + public void createTable() throws SQLException { + StringBuilder buffer = new StringBuilder(); + buffer.append("CREATE TABLE ").append(name); + buffer.append(" (key ").append(getSQLType(keyType)).append(" UNIQUE,value "); + buffer.append(getSQLType(valueType)).append(')'); + String sqlstring = buffer.toString(); + Statement st = getReusableStatement(); + st.executeUpdate(sqlstring); + sqlstring = generateSQLCommand(GRANT_STMT); + st.executeUpdate(sqlstring); + } + + /** + * Clear all rows from the table + * @throws SQLException for problems with the connection + */ + public void clearTable() throws SQLException { + String sqlString = generateSQLCommand(DELETE_ALL_STMT); + getReusableStatement().executeUpdate(sqlString); + } + + /** + * Determine whether a given table exists in the database + * @return true is the table exists + * @throws SQLException for problems with the connection + */ + public boolean exists() throws SQLException { + String sqlString = generateSQLCommand(TABLE_EXISTS_STMT); + boolean result = false; + try (ResultSet rs = getReusableStatement().executeQuery(sqlString)) { + if (rs.next()) { + result = rs.getString(1).equals("public"); + } + } + return result; + } + + /** + * Given a key, retrieve the corresponding value + * @param key identifies the table row + * @return the value corresponding to the key + * @throws SQLException for problems with the connection + */ + public Object readValue(Object key) throws SQLException { + PreparedStatement s = + selectStatement.prepareIfNeeded(() -> db.prepareStatement(SELECT_STMT)); + Object value = null; + s.setObject(1, key, keyType); + try (ResultSet rs = s.executeQuery()) { + if (rs.next()) { + value = rs.getObject(1); + } + } + return value; + } + + /** + * Associate a new value with a given key + * @param key identifies the table row + * @param value is stored at that row + * @throws SQLException for problems with the connection + */ + public void writeValue(Object key, Object value) throws SQLException { + PreparedStatement s = + updateStatement.prepareIfNeeded(() -> db.prepareStatement(UPDATE_STMT)); + s.setObject(1, value, valueType); + s.setObject(2, key, keyType); + + if (s.executeUpdate() == 1) { + return; // Update was successful + } + + s = insertStatement.prepareIfNeeded(() -> db.prepareStatement(INSERT_STMT)); + s.setObject(1, key, keyType); + s.setObject(2, value, valueType); + s.executeUpdate(); + } + + /** + * Deletes the row corresponding to a given key + * @param key identifies the table row + * @throws SQLException for problems with the connection + */ + public void deleteValue(Object key) throws SQLException { + PreparedStatement s = + deleteStatement.prepareIfNeeded(() -> db.prepareStatement(DELETE_STMT)); + s.setObject(1, key, keyType); + s.executeUpdate(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/SQLComplexTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/SQLComplexTable.java new file mode 100755 index 0000000000..06ff18d828 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/SQLComplexTable.java @@ -0,0 +1,97 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; + +import javax.help.UnsupportedOperationException; + +public abstract class SQLComplexTable { + + protected final String tableName; + protected final String idColumnName; // may be null if not applicable + + protected Connection db = null; + + private final CachedStatement deleteStatement; + + public SQLComplexTable(String tableName, String idColumnName) { + this.tableName = tableName; + this.idColumnName = idColumnName; + deleteStatement = idColumnName != null ? new CachedStatement<>() : null; + } + + public void setConnection(Connection db) { + this.db = db; + } + + public void close() { + if (deleteStatement != null) { + deleteStatement.close(); + } + db = null; + } + + /** + * Creates the db table. + * + * @param st the query statement + * @throws SQLException if there is a problem + */ + public abstract void create(Statement st) throws SQLException; + + /** + * Deletes the row with the given id from the db. Users must set the {@link #DELETE_STMT} string + * to delete the exact table they need. + * + * @param id the database row ID + * @return the number of deleted rows + * @throws SQLException if there is a problem creating or executing the query + */ + public int delete(long id) throws SQLException { + if (idColumnName == null) { + throw new UnsupportedOperationException("delete not supported without id column"); + } + PreparedStatement s = deleteStatement.prepareIfNeeded(() -> db + .prepareStatement("DELETE FROM " + tableName + " WHERE " + idColumnName + " = ?")); + s.setLong(1, id); + return s.executeUpdate(); + } + + /** + * Drops the current table. + * NOTE: If explicitly created index tables exist they should be removed first + * or this method override. + * + * @param st the query statement + * @throws SQLException if there is a problem with the execute update command + */ + public void drop(Statement st) throws SQLException { + String sql = "DROP TABLE " + tableName; + st.executeUpdate(sql); + } + + /** + * Inserts a row(s) into the db. The arguments passed to this function are by definition + * not known, so they are left as a variable-length list of {@link Object} instances, to be + * interpreted by the implementer. + * + * @param arguments any arguments required for the insert + * @return to be defined by the implementor + * @throws SQLException if there is a problem executing the insert command + */ + public abstract long insert(Object... arguments) throws SQLException; +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/SQLStringTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/SQLStringTable.java new file mode 100755 index 0000000000..3a318d6e74 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/SQLStringTable.java @@ -0,0 +1,269 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; +import java.util.TreeMap; + +public class SQLStringTable { + public static class StringRecord { + public long id; // Row id of the string record + public String value; // The actual string + public StringRecord prev; + public StringRecord next; + } + + private final String name; // Name of the SQL table + private final int maxLoaded; // Maximum number of string in memory at once + + private final String insertSQL; + private final String selectByIdSQL; + private final String selectByValueSQL; + + private final CachedStatement insertStatement; + private final CachedStatement selectByIdStatement; + private final CachedStatement selectByValueStatement; + + private final TreeMap stringMap; + private final TreeMap idMap; + + private StringRecord loadedHead; // Head of linked list + private StringRecord loadedTail; // End of linked list + + private Connection db; + + public SQLStringTable(String name, int maxloaded) { + this.name = name; + this.maxLoaded = maxloaded; + stringMap = new TreeMap(); + idMap = new TreeMap(); + + insertSQL = generateSQLCommand("INSERT INTO # (id,val) VALUES(DEFAULT,?)"); + selectByIdSQL = generateSQLCommand("SELECT val FROM # WHERE id = ?"); + selectByValueSQL = generateSQLCommand("SELECT id FROM # WHERE val = ?"); + + insertStatement = new CachedStatement<>(); + selectByIdStatement = new CachedStatement<>(); + selectByValueStatement = new CachedStatement<>(); + } + + public void setConnection(Connection db) { + this.db = db; + } + + public void close() { + insertStatement.close(); + selectByIdStatement.close(); + selectByValueStatement.close(); + db = null; // We do not own the db + if (stringMap != null) { + stringMap.clear(); + } + if (idMap != null) { + idMap.clear(); + } + loadedHead = null; + loadedTail = null; + } + + /** + * Create this specific table in the database + * @throws SQLException if the create statement fails + */ + public void createTable() throws SQLException { + String sqlstring = + generateSQLCommand("CREATE TABLE # (id SERIAL PRIMARY KEY,val TEXT UNIQUE)"); + Statement st = db.createStatement(); + st.executeUpdate(sqlstring); + st.close(); + } + + /** + * Try to fetch string from our memory cache, or load it from database, or return empty string + * + * @param id the row ID + * @return the string fetched from the table, or empty string if not found + * @throws SQLException if there is a problem parsing the table record(s) + */ + public String getString(long id) throws SQLException { + if (id == 0) { + return null; + } + StringRecord rec = idMap.get((int) id); + if (rec != null) { + moveToEnd(rec); + return rec.value; + } + String res = readStringRecord(id); + if (res == null) { + throw new SQLException("Id is not present in string table: " + name); + } + return res; + } + + public long writeString(String val) throws SQLException { + if ((val == null) || (val.length() == 0)) { + return 0; + } + StringRecord rec = stringMap.get(val); + if (rec != null) { + moveToEnd(rec); + return rec.id; + } + // String is not in cache + long id = readStringId(val); // Try to read from database + if (id != 0) { + return id; + } + return writeNewString(val); // Otherwise add the string to the table + } + + private void insertAtEnd(StringRecord rec) { + if (loadedTail == null) { + loadedHead = rec; + loadedHead.prev = null; + loadedHead.next = null; + loadedTail = rec; + return; + } + rec.prev = loadedTail; + loadedTail.next = rec; + rec.next = null; + loadedTail = rec; + } + + private StringRecord popFirst() { + StringRecord res = loadedHead; + loadedHead = loadedHead.next; + if (loadedHead == null) { // List is now empty + loadedTail = null; + return res; + } + loadedHead.prev = null; + return res; + } + + private void moveToEnd(StringRecord rec) { + if (rec.next == null) { + return; // Already at end + } + StringRecord prev = rec.prev; + rec.next.prev = prev; + if (prev == null) { + loadedHead = rec.next; + } + else { + prev.next = rec.next; + } + insertAtEnd(rec); + } + + /** + * Purge the top element of the -loaded- list + */ + private void purgeString() { + StringRecord rec = popFirst(); + stringMap.remove(rec.value); + idMap.remove((int) rec.id); + } + + /** + * Replace '#' character in the -ptr- command with our -name- and return the string + * @param ptr is the string to be modified + * @return the modified string + */ + private String generateSQLCommand(String ptr) { + int first = ptr.indexOf('#'); + int second = ptr.indexOf('#', first + 1); + String res; + res = ptr.substring(0, first); + res += name; + if (second >= 0) { + res += ptr.substring(first + 1, second); + res += name; + res += ptr.substring(second + 1); + } + else { + res += ptr.substring(first + 1); + } + return res; + } + + private void insertRecord(long id, String value) { + while (idMap.size() >= maxLoaded) { + purgeString(); + } + StringRecord rec = new StringRecord(); + insertAtEnd(rec); + rec.id = id; + rec.value = value; + stringMap.put(value, rec); + idMap.put(Integer.valueOf((int) id), rec); + } + + private String readStringRecord(long id) throws SQLException { + + PreparedStatement s = + selectByIdStatement.prepareIfNeeded(() -> db.prepareStatement(selectByIdSQL)); + String value = null; + s.setInt(1, (int) id); + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { + return value; + } + value = rs.getString(1); + } + insertRecord(id, value); + return value; + } + + /** + * Try to read the id of a specific string in the table + * @param value is the string to try to find + * @return the id of the string or 0 if the string is not in the table + * @throws SQLException if the result set cannot be parsed + */ + public long readStringId(String value) throws SQLException { + PreparedStatement s = + selectByValueStatement.prepareIfNeeded(() -> db.prepareStatement(selectByValueSQL)); + long id = 0; + s.setString(1, value); + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { + return id; + } + id = rs.getInt(1); + } + insertRecord(id, value); + return id; + } + + private long writeNewString(String value) throws SQLException { + PreparedStatement s = insertStatement.prepareIfNeeded( + () -> db.prepareStatement(insertSQL, Statement.RETURN_GENERATED_KEYS)); + s.setString(1, value); + s.executeUpdate(); + try (ResultSet rs = s.getGeneratedKeys()) { + if (!rs.next()) { + throw new SQLException("Error during insertion"); + } + long id = rs.getInt(1); + insertRecord(id, value); + return id; + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/StatementSupplier.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/StatementSupplier.java new file mode 100644 index 0000000000..7be05c3218 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/StatementSupplier.java @@ -0,0 +1,34 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.SQLException; +import java.sql.Statement; + +/** + * {@link StatementSupplier} provides a callback function to generate a {@link Statement}. + * + * @param {@link Statement} implementation class + */ +public interface StatementSupplier { + + /** + * Return a {@link Statement} for use within the current thread. + * @return statement + * @throws SQLException if callback fails when producing the statement + */ + S get() throws SQLException; +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/WeightTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/WeightTable.java new file mode 100755 index 0000000000..75d8ea21ab --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/client/tables/WeightTable.java @@ -0,0 +1,83 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client.tables; + +import java.sql.*; + +import generic.lsh.vector.WeightFactory; + +public class WeightTable extends SQLComplexTable { + + public WeightTable() { + super("weighttable", "id"); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate("CREATE TABLE weighttable(id integer,weight NUMERIC(24,20))"); + } + + @Override + public void drop(Statement st) throws SQLException { + String sql = "DROP TABLE IF EXISTS " + tableName; + st.executeUpdate(sql); + } + + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 2) { + throw new IllegalArgumentException( + "Insert method for WeightTable must take exactly two arguments: int and double"); + } + + final Statement st = db.createStatement(); + + final int row = (int) arguments[0]; + final double val = (double) arguments[1]; + + final StringBuffer buf = new StringBuffer(); + buf.append("INSERT INTO weighttable (id,weight) VALUES("); + buf.append(row).append(',').append(val).append(')'); + st.executeUpdate(buf.toString()); + + return 0; + } + + /** + * + * @param factory the weight factory + * @throws SQLException if there is an error creating/executing the query + */ + public void recoverWeights(WeightFactory factory) throws SQLException { + try (Statement st = db.createStatement(); + ResultSet rs = st.executeQuery("SELECT all * FROM weighttable")) { + double vals[] = new double[factory.getIDFSize() + factory.getTFSize() + 7]; + int numrows = 0; + while (rs.next()) { + int id = rs.getInt(1); + double val = rs.getDouble(2); + vals[id] = val; + numrows += 1; + } + if (numrows != factory.getIDFSize() + factory.getTFSize() + 7) { + throw new SQLException("weighttable has wrong number of rows"); + } + + factory.set(vals); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/CallgraphEntry.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/CallgraphEntry.java new file mode 100755 index 0000000000..0c81126a68 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/CallgraphEntry.java @@ -0,0 +1,132 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +import java.io.IOException; +import java.io.Writer; + +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +public class CallgraphEntry implements Comparable { + private FunctionDescription dest; // Function being called + private int lochash; // Location hash of callsite + + public CallgraphEntry(FunctionDescription d,int lhash) { + dest = d; + lochash = lhash; + } + + public FunctionDescription getFunctionDescription() { return dest; } + + public int getLocalHash() { return lochash; } + + public void saveXml(FunctionDescription src,Writer fwrite) throws IOException { + StringBuilder buf = new StringBuilder(); + buf.append("\n"); + if (!destexe.isLibrary()) { + buf.append(" ").append(destexe.getMd5()).append("\n"); + } + buf.append(" "); + SpecXmlUtils.xmlEscape(buf, destexe.getNameExec()); + buf.append("\n"); + if (!srcexe.getArchitecture().equals(destexe.getArchitecture())) { + buf.append(" "); + SpecXmlUtils.xmlEscape(buf, destexe.getArchitecture()); + buf.append("\n"); + } + if (!srcexe.getNameCompiler().equals(destexe.getNameCompiler())) { + buf.append(" "); + SpecXmlUtils.xmlEscape(buf, destexe.getNameCompiler()); + buf.append("\n"); + } + buf.append("\n"); + } + else { + buf.append("/>\n"); + } + fwrite.append(buf.toString()); + } + + static public void restoreXml(XmlPullParser parser,DescriptionManager man,FunctionDescription src) throws LSHException { + XmlElement el = parser.start("call"); + String destnm = el.getAttribute("dest"); + long address = -1; // Default if no "addr" attribute present + String addrString = el.getAttribute("addr"); + if (addrString != null) { + address = SpecXmlUtils.decodeLong(addrString); + } + int val = SpecXmlUtils.decodeInt(el.getAttribute("local")); + if (parser.peek().isStart()) { + ExecutableRecord srcexe = src.getExecutableRecord(); + String md5 = null; + String dest_enm = null; + String dest_cnm = srcexe.getNameCompiler(); + String dest_arch = srcexe.getArchitecture(); + do { + String elname = parser.next().getName(); + String content = parser.end().getText(); + if (elname.equals("md5")) { + md5 = content; + } + else if (elname.equals("name")) { + dest_enm = content; + } + else if (elname.equals("compiler")) { + dest_cnm = content; + } + else if (elname.equals("arch")) { + dest_arch = content; + } + } while(parser.peek().isStart()); + if (md5 == null) { + ExecutableRecord destexe = man.newExecutableLibrary(dest_enm,dest_arch,null); + FunctionDescription destfunc = man.newFunctionDescription(destnm, address, destexe); + man.makeCallgraphLink(src, destfunc, val); + } + else { + ExecutableRecord destexe = man.newExecutableRecord(md5, dest_enm, dest_cnm, dest_arch, null, srcexe.getRepository(), srcexe.getPath(), null); + FunctionDescription destfunc = man.newFunctionDescription(destnm, address, destexe); + man.makeCallgraphLink(src, destfunc, val); + } + } + else { // Assume dest is in same executable as src + FunctionDescription destfunc = + man.newFunctionDescription(destnm, address, src.getExecutableRecord()); + man.makeCallgraphLink(src, destfunc, val); + } + parser.end(); + } + + @Override + public int compareTo(CallgraphEntry o) { + return dest.compareTo(o.dest); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/CategoryRecord.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/CategoryRecord.java new file mode 100755 index 0000000000..c18716a75c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/CategoryRecord.java @@ -0,0 +1,98 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +import java.io.IOException; +import java.io.Writer; + +/** + * A user-defined category associated associated with an executable + * Specified by a -type- and then the particular -category- (within the type) that + * the executable belongs to. + * + */ +public class CategoryRecord implements Comparable { + + private String type; // The type of category (must not be null) + private String category; // The type specific category + + public CategoryRecord(String t,String c) { + type = t; + category = c; + } + + public String getType() { + return type; + } + + public String getCategory() { + return category; + } + + @Override + public boolean equals(Object obj) { + CategoryRecord op2 = (CategoryRecord)obj; + if (!type.equals(op2.type)) return false; + return category.equals(op2.category); + } + + @Override + public int compareTo(CategoryRecord arg0) { + int cmp = type.compareTo(arg0.type); + if (cmp != 0) + return cmp; + if (category == null) { + if (arg0.category == null) return 0; + return -1; // this precedes anything non-null + } + if (arg0.category==null) + return 1; // this comes after null + return category.compareTo(arg0.category); + } + + public void saveXml(Writer fwrite) throws IOException { + fwrite.append(" "); + SpecXmlUtils.xmlEscapeWriter(fwrite, category); + fwrite.append("\n"); + } + + public static CategoryRecord restoreXml(XmlPullParser parser) throws LSHException { + XmlElement el = parser.start("category"); + String type = el.getAttribute("type"); + String category = parser.end().getText(); + if (type==null || category==null) + throw new LSHException("Bad category tag"); + return new CategoryRecord(type,category); + } + + public static boolean enforceTypeCharacters(String val) { + if (val==null) return false; + if (val.length()==0) return false; + for(int i=0;i execats; // Executable categories for this database + public List functionTags; // Named boolean properties on functions + public String dateColumnName; // An override of the name "Ingest Date" + public int layout_version; // Version of the database layout + public boolean readonly; // -true- if database is readonly + public boolean trackcallgraph; // -true- if database tracks callgraph information of executables + + public DatabaseInformation() { + databasename = "Example Database"; + owner = "Example Owner"; + description = "A collection of functions for testing purposes"; + major = 0; // A zero major version indicates no data has been inserted yet + minor = 0; + settings = 0; + execats = null; + functionTags = null; + dateColumnName = null; + layout_version = 0; + readonly = false; + trackcallgraph = true; + } + + public void saveXml(Writer write) throws IOException { + write.append("\n"); + if (databasename != null) + write.append(" ").append(databasename).append("\n"); + else + write.append(" \n"); + if (owner != null) + write.append(" ").append(owner).append("\n"); + else + write.append(" \n"); + if (description != null) + write.append(" ").append(description).append("\n"); + else + write.append(" \n"); + write.append(" ").append(Short.toString(major)).append("\n"); + write.append(" ").append(Short.toString(minor)).append("\n"); + write.append(" 0x").append(Integer.toHexString(settings)).append("\n"); + if (execats != null) { + for (String cat : execats) + write.append(" ").append(cat).append("\n"); + } + if (functionTags != null) { + for (String tag : functionTags) + write.append(" ").append(tag).append("\n"); + } + if (dateColumnName != null) { + write.append(" ").append(dateColumnName).append("\n"); + } + if (readonly) + write.append(" true\n"); + if (!trackcallgraph) + write.append(" false\n"); + write.append(" ").append(Integer.toString(layout_version)).append("\n"); + write.append("\n"); + } + + public void restoreXml(XmlPullParser parser) { + parser.start("info"); + parser.start("name"); + databasename = parser.end().getText(); + if (databasename.length() == 0) + databasename = null; + parser.start("owner"); + owner = parser.end().getText(); + if (owner.length() == 0) + owner = null; + parser.start("description"); + description = parser.end().getText(); + if (description.length() == 0) + description = null; + parser.start("major"); + major = (short) SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("minor"); + minor = (short) SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("settings"); + settings = SpecXmlUtils.decodeInt(parser.end().getText()); + readonly = false; + trackcallgraph = true; + layout_version = 0; + execats = null; + functionTags = null; + dateColumnName = null; + while (parser.peek().isStart()) { + XmlElement el = parser.start(); + if (el.getName().equals("readonly")) + readonly = SpecXmlUtils.decodeBoolean(parser.end().getText()); + else if (el.getName().equals("trackcallgraph")) + trackcallgraph = SpecXmlUtils.decodeBoolean(parser.end().getText()); + else if (el.getName().equals("layout")) + layout_version = SpecXmlUtils.decodeInt(parser.end().getText()); + else if (el.getName().equals("execategory")) { + if (execats == null) + execats = new ArrayList(); + String cat = parser.end().getText(); + execats.add(cat); + } + else if (el.getName().equals("functiontag")) { + if (functionTags == null) + functionTags = new ArrayList(); + String tag = parser.end().getText(); + functionTags.add(tag); + } + else if (el.getName().equals("datename")) { + dateColumnName = parser.end().getText(); + } + } + parser.end(); + } + + @Override + public boolean equals(Object obj) { + // FIXME - missing hashcode method - is equals really used? + if (true) { + throw new AssertException( + "DatabaseInformation.equals is used - should add hashcode method"); + } + DatabaseInformation op2 = (DatabaseInformation) obj; + if (!op2.databasename.equals(databasename)) + return false; + if (op2.major != major) + return false; + if (op2.minor != minor) + return false; + if (op2.settings != settings) + return false; + if (op2.execats == null) { + if (execats != null) + return false; + } + else { + if (execats == null) + return false; + if (op2.execats.size() != execats.size()) + return false; + for (int i = 0; i < execats.size(); ++i) { + if (!op2.execats.get(i).equals(execats.get(i))) + return false; + } + } + if (op2.functionTags == null) { + if (functionTags != null) + return false; + } + else { + if (functionTags == null) + return false; + if (op2.functionTags.size() != functionTags.size()) + return false; + for (int i = 0; i < functionTags.size(); ++i) { + if (!op2.functionTags.get(i).equals(functionTags.get(i))) + return false; + } + } + if (op2.dateColumnName == null) { + if (dateColumnName != null) + return false; + } + else { + if (dateColumnName == null) + return false; + if (!op2.dateColumnName.equals(dateColumnName)) + return false; + } + if (op2.layout_version != layout_version) + return false; + if (op2.readonly != readonly) + return false; + if (op2.trackcallgraph != trackcallgraph) + return false; + return true; + } + + public int checkSignatureSettings(short maj, short min, int set) { + if ((maj == 0) || (set == 0)) + return 3; // No setting information + if ((major == 0) || (settings == 0)) + return 4; // This has no setting information + if ((major != maj) || (settings != set)) + return 2; // There is a setting mismatch, major version and settings must match + if (minor == min) + return 0; // There is a complete settings match + if (minor > min) { + if (minor - min > 1) + return 2; // Settings mismatch (minor versions differ too much) + } + else { + if (min - minor > 1) + return 2; // Settings mismatch + } + return 1; // Only a minor difference in version and settings + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/DescriptionManager.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/DescriptionManager.java new file mode 100755 index 0000000000..c6ad517abd --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/DescriptionManager.java @@ -0,0 +1,697 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import org.apache.commons.lang3.StringUtils; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Container for metadata about executables (ExecutableRecord), + * functions (FunctionDescription) and their associated signatures (SignatureRecord) + * Generally holds sets of functions that are either being inserted into + * are queried from a BSim database + */ +public class DescriptionManager { + public static final int LAYOUT_VERSION = 5; // This versions the XML serialization of the objects put in a DescriptionManager + + private TreeSet funcrec; // Functions in this container (sort by exe,name,address) + private TreeSet exerec; // Executables in this container (sort by md5) + private TreeMap rowCache; // Alternate index into executables via row id + private short major; // Major version of decompiler used to generate SignatureRecords + private short minor; // Minor version + private int settings; // Settings for signature generation (of functions in this container) + + public DescriptionManager() { + funcrec = new TreeSet<>(); + exerec = new TreeSet<>(); + rowCache = null; + } + + /** + * Set the version number of the decompiler used to generate SignatureRecords + * for this container + * @param maj is the major number + * @param min is the minor + */ + public void setVersion(short maj, short min) { + major = maj; + minor = min; + } + + /** + * Establish the particular settings of the signature strategy used to + * generate SignatureRecords for this container + * @param set is the encoded bit-field of settings + */ + public void setSettings(int set) { + settings = set; + } + + /** + * @return the major version number of the decompiler used for signatures + */ + public short getMajorVersion() { + return major; + } + + /** + * @return the minor version number of the decompiler used for signatures + */ + public short getMinorVersion() { + return minor; + } + + /** + * @return the settings of the signature strategy used for this container + */ + public int getSettings() { + return settings; + } + + /** + * Set the categories associated with a particular executable. + * This replaces any existing categories + * @param erec is the ExecutableRecord to set + * @param cats is the list of categories (CategoryRecord), may be null + */ + public void setExeCategories(ExecutableRecord erec, List cats) { + erec.setCategory(cats); + } + + /** + * Associate a database id with a particular executable + * @param erec is the ExecutableRecord + * @param id is the database (row) id + */ + public void setExeRowId(ExecutableRecord erec, RowKey id) { + erec.setRowId(id); + } + + /** + * Mark that an executable has (already) been stored in the database + * @param erec is the ExecutableRecord + */ + public void setExeAlreadyStored(ExecutableRecord erec) { + erec.setAlreadyStored(); + } + + /** + * Associate a signature's id with a particular function + * @param frec is the FunctionDescription + * @param id is the signature's database id + */ + public void setSignatureId(FunctionDescription frec, long id) { + frec.setVectorId(id); + } + + /** + * Associate a database id with a particular SignatureRecord + * @param sigrec is the SignatureRecord + * @param id is the signature's database id + */ + public void setSignatureId(SignatureRecord sigrec, long id) { + sigrec.setVectorId(id); + } + + /** + * Associate a database id with a particular function + * @param fd is the FunctionDescription + * @param id is the database (row) id + */ + public void setFunctionDescriptionId(FunctionDescription fd, RowKey id) { + fd.setId(id); + } + + /** + * Associate function "tags" or attributes with a specific function + * @param fd is the FunctionDescription + * @param fl is the encoded bitfield of attributes + */ + public void setFunctionDescriptionFlags(FunctionDescription fd, int fl) { + fd.setFlags(fl); + } + + public TreeSet getExecutableRecordSet() { + return exerec; + } + + /** + * Clear out all functions from the container, but leave the executables + */ + public void clearFunctions() { + funcrec.clear(); + } + + /** + * Reset to a completely empty container + */ + public void clear() { + clearFunctions(); + major = 0; + minor = 0; + settings = 0; + exerec.clear(); + rowCache = null; + } + + /** + * @return the number of executables described by this container + */ + public int numExecutables() { + return exerec.size(); + } + + /** + * @return the number of functions described by this container + */ + public int numFunctions() { + return funcrec.size(); + } + + /** + * Allocate a new function in the container + * @param fnm is the name of the new function + * @param address is the address (offset) of the function + * @param erec is the executable containing the function + * @return the new FunctionDescription + */ + public FunctionDescription newFunctionDescription(String fnm, long address, + ExecutableRecord erec) { + FunctionDescription newfunc = new FunctionDescription(erec, fnm, address); + if (!funcrec.add(newfunc)) { + newfunc = funcrec.floor(newfunc); + } + return newfunc; + } + + /** + * Create a new executable record, which should be identified uniquely + * identified via its md5sum + * + * @param md5 is the MD5 hash of the executable + * @param enm is the name of the executable + * @param cnm is the name of the compiler used to build the executable + * @param arc is the architecture of the executable + * @param dt is the date (of ingest) + * @param repo is the repository containing the executable + * @param path is the path (within the repo) to the executable + * @param id is the database (row) is associated with the executable (may be null) + * @return the new ExecutableRecord object + * @throws LSHException if attributes are invalid, or the executable + * already exists with different metadata + */ + public ExecutableRecord newExecutableRecord(String md5, String enm, String cnm, String arc, + Date dt, String repo, String path, RowKey id) throws LSHException { + if (md5.length() != 32) { + throw new LSHException("MD5 field must be exactly 32 hex characters"); + } + ExecutableRecord newexe = new ExecutableRecord(md5, enm, cnm, arc, dt, id, repo, path); + if (!exerec.add(newexe)) { + ExecutableRecord oldexe = exerec.floor(newexe); + if (oldexe.compareMetadata(newexe) != 0) { + throw new LSHException("Duplicate md5 hash, different metadata"); + } + if ((oldexe.getRowId() != null) && (id != null) && (!oldexe.getRowId().equals(id))) { + throw new LSHException("Overwriting existing executable id"); + } + newexe = oldexe; + } + return newexe; + } + + /** + * Create a new "library" executable in the container. + * Functions in this container (will) have no body or address + * @param enm is the name of the library + * @param arc is the architecture of the library + * @param id is the database id associated with the library (may be null) + * @return the new ExecutableRecord object + * @throws LSHException if attributes are invalid or the + * library already exists with different metadata + */ + public ExecutableRecord newExecutableLibrary(String enm, String arc, RowKey id) + throws LSHException { + ExecutableRecord newexe = new ExecutableRecord(enm, arc, id); + if (!exerec.add(newexe)) { // Check for duplicate executable + ExecutableRecord oldexe = exerec.floor(newexe); + if (oldexe.compareMetadata(newexe) != 0) { + throw new LSHException("Duplicate md5 hash, different metadata"); + } + if ((oldexe.getRowId() != null) && (id != null) && (!oldexe.getRowId().equals(id))) { + throw new LSHException("Overwriting existing executable id"); + } + newexe = oldexe; + } + return newexe; + } + + /** + * Transfer decompiler and signature settings into this container + * @param op2 is the container to transfer from + */ + public void transferSettings(DescriptionManager op2) { + major = op2.major; + minor = op2.minor; + settings = op2.settings; + } + + /** + * Transfer an executable from another container into this container + * @param erec is the ExecutableRecord from the other container + * @return the new transferred ExecutableRecord + * @throws LSHException if the executable already exists with different metadata + */ + public ExecutableRecord transferExecutable(ExecutableRecord erec) throws LSHException { + RowKey id = erec.getRowId(); + + ExecutableRecord res; + if (erec.isLibrary()) { + res = newExecutableLibrary(erec.getNameExec(), erec.getArchitecture(), id); + } + else { + res = newExecutableRecord(erec.getMd5(), erec.getNameExec(), erec.getNameCompiler(), + erec.getArchitecture(), (Date) erec.getDate().clone(), erec.getRepository(), + erec.getPath(), id); + } + res.cloneCategories(erec); + return res; + } + + /** + * Transfer a function from another container into this container + * @param fdesc is the FunctionDescription to transfer + * @param transsig is true if the SignatureRecord should be transferred as well + * @return the new transferred FunctionDescription + * @throws LSHException if the function already exists with different metadata + */ + public FunctionDescription transferFunction(FunctionDescription fdesc, boolean transsig) + throws LSHException { + ExecutableRecord erec = transferExecutable(fdesc.getExecutableRecord()); + FunctionDescription res = + newFunctionDescription(fdesc.getFunctionName(), fdesc.getAddress(), erec); + res.setVectorId(fdesc.getVectorId()); + res.setFlags(fdesc.getFlags()); + SignatureRecord srec = fdesc.getSignatureRecord(); + if (transsig && (srec != null)) { + SignatureRecord sigclone = newSignature(srec.getLSHVector(), srec.getCount()); + attachSignature(res, sigclone); + } + return res; + } + + /** + * Generate a map from (row) id to function, for all functions in this container + * @param funcmap is the map to populate + */ + public void generateFunctionIdMap(Map funcmap) { + for (FunctionDescription func : funcrec) { + funcmap.put(func.getId(), func); + } + } + + /** + * Generate a SignatureRecord given a specific feature vector + * @param vec is the feature vector (LSHVector) + * @param count is a count of functions sharing this feature vector + * @return the new SignatureRecord + */ + public SignatureRecord newSignature(LSHVector vec, int count) { + SignatureRecord srec = new SignatureRecord(vec); + srec.setCount(count); + return srec; + } + + /** + * Parse a signature (SignatureRecord) from an XML stream + * @param parser is the XML parser + * @param vectorFactory is the factory used to generate the underlying feature vector + * @param count is the count of functions sharing the feature vector + * @return the new SignatureRecord + */ + public SignatureRecord newSignature(XmlPullParser parser, LSHVectorFactory vectorFactory, + int count) { + LSHVector res = vectorFactory.restoreVectorFromXml(parser); + SignatureRecord srec = new SignatureRecord(res); + + srec.setCount(count); + return srec; + } + + /** + * Associate a signature with a specific function + * @param fd is the FunctionDescription + * @param srec is the SignatureRecord + */ + public void attachSignature(FunctionDescription fd, SignatureRecord srec) { + fd.setSignatureRecord(srec); + setSignatureId(fd, srec.getVectorId()); + } + + /** + * Mark a parent/child relationship between to functions + * @param src is the parent FunctionDescription + * @param dest is the child FunctionDescription + * @param lhash is a hash indicating where in -src- the call to -dest- is made + */ + public void makeCallgraphLink(FunctionDescription src, FunctionDescription dest, int lhash) { + src.insertCall(dest, lhash); + } + + /** + * Lookup an executable in the container via md5 + * @param md5 is the md5 to search for + * @return return the matching ExecutableRecord + * @throws LSHException if the executable cannot be found + */ + public ExecutableRecord findExecutable(String md5) throws LSHException { + ExecutableRecord templ = new ExecutableRecord(md5); + ExecutableRecord res = exerec.floor(templ); + if (res != null && res.getMd5().equals(md5)) { + return res; + } + throw new LSHException("Unable to find executable"); + } + + /** + * Search for executable based an name, and possibly other qualifying information. + * This is relatively inefficient as it just iterates through the list. + * @param name is the name that the executable must match + * @param arch is null or must match the executable's architecture string + * @param comp is null or must match the executable's compiler string + * @return the matching executable + * @throws LSHException if a matching executable doesn't exist + */ + public ExecutableRecord findExecutable(String name, String arch, String comp) + throws LSHException { + if (StringUtils.isEmpty(arch)) { + arch = null; + } + if (StringUtils.isEmpty(comp)) { + comp = null; + } + for (ExecutableRecord erec : exerec) { + if (!erec.getNameExec().equals(name)) { + continue; + } + if (arch != null && !erec.getArchitecture().equals(arch)) { + continue; + } + if (comp != null && !erec.getNameCompiler().equals(comp)) { + continue; + } + return erec; + } + throw new LSHException("Unable to find executable"); + } + + /** + * Find a function (within an executable) by its name and address (both must be provided) + * If the request function does not exist, an exception is thrown + * @param fname - the name of the function + * @param address - the address of the function + * @param exe - the ExecutableRecord containing the function + * @return the FunctionDescription + * @throws LSHException if a matching function does not exist + */ + public FunctionDescription findFunction(String fname, long address, ExecutableRecord exe) + throws LSHException { + FunctionDescription fdesc = new FunctionDescription(exe, fname, address); + + FunctionDescription res = funcrec.floor(fdesc); + if (res == null || (!res.equals(fdesc))) { + throw new LSHException("Unable to find FunctionDescription"); + } + return res; + } + + /** + * Find a function within an executable by name. The name isn't guaranteed to be unique. If there + * are more than one, the first in address order is returned. If none are found, null is returned + * @param fname is the name of the function to match + * @param exe is the ExecutableRecord containing the function + * @return a FunctionDescription or null + */ + public FunctionDescription findFunctionByName(String fname, ExecutableRecord exe) { + FunctionDescription fdesc = new FunctionDescription(exe, fname, 0); + FunctionDescription res = funcrec.ceiling(fdesc); + if (res == null || !fname.equals(res.getFunctionName()) || + !res.getExecutableRecord().equals(exe)) { + return null; + } + return res; + } + + /** + * Find a function (within an executable) by its name and address (both must be provided) + * If the function doesn't exist, null is returned, no exception is thrown + * @param fname - the name of the function + * @param address - the address of the function + * @param exe - the executable (possibly) containing the function + * @return a FunctionDescription or null + */ + public FunctionDescription containsDescription(String fname, long address, + ExecutableRecord exe) { + FunctionDescription fdesc = new FunctionDescription(exe, fname, address); + FunctionDescription res = funcrec.floor(fdesc); + if (res == null || (!res.equals(fdesc))) { + return null; + } + return res; + } + + /** + * Generate an iterator over all functions belonging to a specific executable + * @param exe is the specific executable + * @return iterator over all functions in -exe- + */ + public Iterator listFunctions(ExecutableRecord exe) { + ExecutableRecord startexe = exerec.floor(exe); + ExecutableRecord endexe = exerec.higher(exe); + if (startexe == null) { + return null; + } + FunctionDescription startfunc = new FunctionDescription(startexe, "", 0); + startfunc = funcrec.ceiling(startfunc); + if (startfunc == null) { // No functions in exe or after + startfunc = funcrec.last(); + return funcrec.subSet(startfunc, startfunc).iterator(); + } + FunctionDescription endfunc = null; + if (endexe != null) { + endfunc = new FunctionDescription(endexe, "", 0); + endfunc = funcrec.ceiling(endfunc); + } + if (endfunc == null) { + // executable is last and has no functions + return funcrec.tailSet(startfunc).iterator(); + } + return funcrec.subSet(startfunc, endfunc).iterator(); + } + + /** + * @return an iterator over all functions in the container + */ + public Iterator listAllFunctions() { + return funcrec.iterator(); + } + + /** + * Using the standard exe-md5, function name, address sorting, return an + * iterator over all functions starting with the first function after + * an indicated -func- + * @param func is FunctionDescription indicating where the iterator should start (after) + * @return the new iterator + */ + public Iterator listFunctionsAfter(FunctionDescription func) { + return funcrec.tailSet(func, false).iterator(); + } + + /** + * Create an internal map entry from a database id to an executable + * @param erec is the ExecutableRecord + * @param rowKey is the database (row) id + */ + public void cacheExecutableByRow(ExecutableRecord erec, RowKey rowKey) { + if (rowCache == null) { + rowCache = new TreeMap<>(); + } + rowCache.put(rowKey, erec); + } + + /** + * Look up an executable via database id. This uses an internal map which + * must have been explicitly populated via cacheExecutableByRow + * @param rowKey is the database (row) id to lookup + * @return the associated ExecutableRecord or null if not found + */ + public ExecutableRecord findExecutableByRow(RowKey rowKey) { + if (rowCache == null) { + return null; + } + return rowCache.get(rowKey); + } + + /** + * Assign an internal id to all executables for purposes of cross-referencing in XML + * Indices are assigned in order starting at 1 (0 indicates an index has NOT been assigned) + */ + public void populateExecutableXref() { + if (exerec.isEmpty()) { + return; + } + if (exerec.first().getXrefIndex() == 1) { + return; // Already been populated + } + int xrefIndex = 1; + for (ExecutableRecord exe : exerec) { + exe.setXrefIndex(xrefIndex); + xrefIndex += 1; + } + } + + /** + * For every ExecutableRecord in this container, if it is also in {@code manage}, + * copy the xrefValue from the {@code manage} version, otherwise + * set the xrefValue to zero + * @param manage is the other container match from + */ + public void matchAndSetXrefs(DescriptionManager manage) { + TreeSet manageSet = manage.exerec; + for (ExecutableRecord currentRecord : exerec) { + ExecutableRecord match = manageSet.floor(currentRecord); + if (match != null && match.getMd5().equals(currentRecord.getMd5())) { + currentRecord.setXrefIndex(match.getXrefIndex()); + } + else { + currentRecord.setXrefIndex(0); // Mark as having no match in manage + } + } + } + + /** + * Assign an internal id to all executables and also create a map from id to executable. + * As with {@link DescriptionManager#populateExecutableXref}, + * ids are assigned in order starting at 1 + * @return the populated Map object + */ + public Map generateExecutableXrefMap() { + TreeMap treeMap = new TreeMap<>(); + int xrefIndex = 1; + for (ExecutableRecord exe : exerec) { + exe.setXrefIndex(xrefIndex); + treeMap.put(xrefIndex, exe); + xrefIndex += 1; + } + return treeMap; + } + + /** + * Override the repository setting of every executable in this manager + * + * @param repo is the repository string to override with + * @param path is the path string to override with + */ + public void overrideRepository(String repo, String path) { + for (ExecutableRecord element : exerec) { + element.setRepository(repo, path); + } + } + + /** + * Serialize the entire container to an XML stream + * @param fwrite is the stream to write to + * @throws IOException if there are problems writing to the stream + */ + public void saveXml(Writer fwrite) throws IOException { + fwrite.append("\n"); + ExecutableRecord curexe = null; + for (FunctionDescription fdesc : funcrec) { + if ((curexe == null) || (0 != fdesc.getExecutableRecord().compareTo(curexe))) { + if (curexe != null) { + fwrite.append("\n"); + } + curexe = fdesc.getExecutableRecord(); + fwrite.append("\n"); + curexe.saveXml(fwrite); + } + fdesc.sortCallgraph(); + fdesc.saveXml(fwrite); + } + if (curexe != null) { + fwrite.append("\n"); + } + fwrite.append("\n"); + } + + /** + * Reconstruct a container by deserializing an XML stream + * @param parser is the XML parser + * @param vectorFactory is the factory to use for building feature vectors + * @throws LSHException if there are inconsistencies in the XML + */ + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + major = 0; + minor = 0; + settings = 0; + int layout_version = 0; + XmlElement el = parser.start("description"); + if (el.hasAttribute("layout_version")) { + layout_version = SpecXmlUtils.decodeInt(el.getAttribute("layout_version")); + } + if (layout_version < LAYOUT_VERSION) { + throw new LSHException("Old XML layout is no longer supported"); + } + if (layout_version > LAYOUT_VERSION) { + throw new LSHException("XML layout for newer version of BSIM"); + } + if (el.hasAttribute("major")) { + major = (short) SpecXmlUtils.decodeInt(el.getAttribute("major")); + minor = (short) SpecXmlUtils.decodeInt(el.getAttribute("minor")); + } + if (el.hasAttribute("settings")) { + settings = SpecXmlUtils.decodeInt(el.getAttribute("settings")); + } + while (parser.peek().isStart()) { + parser.start("execlist"); + ExecutableRecord erec = ExecutableRecord.restoreXml(parser, this); + while (parser.peek().isStart()) { + FunctionDescription.restoreXml(parser, vectorFactory, this, erec); + } + parser.end(); + } + parser.end(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/ExecutableRecord.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/ExecutableRecord.java new file mode 100755 index 0000000000..1860372c62 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/ExecutableRecord.java @@ -0,0 +1,907 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +import java.io.*; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.*; + +import generic.hash.SimpleCRC32; +import ghidra.features.bsim.query.LSHException; +import ghidra.framework.protocol.ghidra.GhidraURL; +import ghidra.util.Msg; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Metadata about a specific executable, as stored in a BSim database + * There are two basic varieties: + * Normal executables, which can be viewed as a container of functions where + * each function has a body and an address (and a corresponding feature vector) + * Library executables, which contains functions that can only be identified by + * name and have no body (or corresponding feature vector) + */ +public class ExecutableRecord implements Comparable { + public static final Date EMPTY_DATE = new Date(0); + + // Boolean attributes associated with an ExecutableRecord via its -flags- field + public static final int ALREADY_STORED = 1; + public static final int LIBRARY = 2; + public static final int CATEGORIES_SET = 4; + + // Flags to indicate differences in metadata + public static final int METADATA_NAME = 1; + public static final int METADATA_ARCH = 2; + public static final int METADATA_COMP = 4; + public static final int METADATA_DATE = 8; + public static final int METADATA_REPO = 16; + public static final int METADATA_PATH = 32; + public static final int METADATA_LIBR = 64; + + private final String md5sum; // The MD5 hash of the executable + private final String executableName; // The name of the executable + private final String architecture; // The architecture on which the executable runs + private final String compilerName; // The name of the compiler used to build the executable + private Date date; // Date of (ingest) + private String repository; // The repository containing the executable + private String path; // The path (within the repository) to the executable + private RowKey rowid; // The primary database key associated with the executable record + private int flags; // Boolean attributes of an executable + private List usercat; // Categories this executable belongs to + private int xrefIndex; // Index for cross-referencing this executable from other records + + public static class Update { + public ExecutableRecord update; + public boolean name_exec; // Should name be updated + public boolean architecture; // Should architecture be updated + public boolean name_compiler; + public boolean repository; + public boolean path; + public boolean date; + public boolean categories; // True if there are either insertions or deletions + public List catinsert; // Non-null, if there are only insertions + } + + /** + * Convert a 32-bit integer to hexadecimal ascii representation + * @param val is the integer to encode + * @param buf accumulates the resulting ascii + */ + private static void wordToAscii(int val, StringBuilder buf) { + for (int i = 28; i >= 0; i -= 4) { + final int nibble = (val >> i) & 0xf; + if (nibble < 10) { + buf.append((char) (nibble + '0')); + } + else { + buf.append((char) (nibble - 10 + 'a')); + } + } + } + + /** + * Generate a placeholder md5 string for a library executable based just + * on its name and architecture + * @param enm is the name of the library + * @param arc is the architecture + * @return the placeholder md5 String + */ + static public String calcLibraryMd5Placeholder(String enm, String arc) { + int hi = 0xb1b110; + int lo = 0xfabafaba; + for (int i = 0; i < enm.length(); ++i) { + final int feed = lo >>> 24; + lo = SimpleCRC32.hashOneByte(lo, enm.charAt(i) & 0xff); + hi = SimpleCRC32.hashOneByte(hi, feed); + } + lo ^= 0xf1b1f1b1; + for (int i = 0; i < arc.length(); ++i) { + final int feed = lo >>> 24; + lo = SimpleCRC32.hashOneByte(lo, arc.charAt(i) & 0xff); + hi = SimpleCRC32.hashOneByte(hi, feed); + } + StringBuilder buf = new StringBuilder(); + buf.append("bbbbbbbbaaaaaaaa"); + wordToAscii(hi, buf); + wordToAscii(lo, buf); + return buf.toString(); + } + + /** + * Constructor for searching within a DescriptionManager + * @param md5 is hash of executable being searched for + */ + protected ExecutableRecord(String md5) { + md5sum = md5; + executableName = ""; + architecture = ""; + compilerName = ""; + rowid = null; + flags = 0; + usercat = null; + xrefIndex = 0; + repository = null; + path = null; + date = EMPTY_DATE; + } + + /** + * Construct a normal (non-library) record. Fill-in all fields except categories. + * Categories are marked as NOT set + * @param md5 is the md5 checksum + * @param execName is the executable name + * @param compilerName is the compiler name + * @param architecture is the processor architecture + * @param date is the date of ingest (may be null) + * @param id is the row id of the record + * @param repo is the repository containing the executable (may be null) + * @param path is the path to the executable (may be null) + */ + public ExecutableRecord(String md5, String execName, String compilerName, String architecture, + Date date, RowKey id, String repo, String path) { + this.md5sum = md5; + this.executableName = execName; + this.architecture = architecture; + this.compilerName = compilerName; + this.rowid = id; + this.flags = 0; + this.usercat = null; + this.xrefIndex = 0; + setRepository(repo, path); + setDate(date); + } + + /** + * Construct a normal (non-library) record. Fill-in all fields. + * @param md5 is the md5 checksum + * @param enm is the executable name + * @param cnm is the compiler name + * @param arc is the architecture + * @param dt is the date of ingest (may be null) + * @param uc is the categories (may be null, categories are considered SET regardless) + * @param id is the row id of the record + * @param repo is the repository containing the executable (may be null) + * @param pth is the path to the executable (may be null) + */ + public ExecutableRecord(String md5, String enm, String cnm, String arc, Date dt, + List uc, RowKey id, String repo, String pth) { + md5sum = md5; + executableName = enm; + architecture = arc; + compilerName = cnm; + rowid = id; + flags = 0; + xrefIndex = 0; + setRepository(repo, pth); + setDate(dt); + setCategory(uc); + } + + /** + * Constructor for a "library" executable + * @param enm is the name of the library + * @param arc is the architecture for functions in the library + * @param id is the database (row) id of the record (may be null) + */ + public ExecutableRecord(String enm, String arc, RowKey id) { + executableName = enm; + architecture = arc; + compilerName = ""; + date = EMPTY_DATE; + repository = null; // Not contained in a repository + path = null; + rowid = id; + flags = LIBRARY; // Indicate that this is a library + md5sum = calcLibraryMd5Placeholder(enm, arc); + usercat = null; + xrefIndex = 0; + } + + /** + * Set the repository and path Strings for an executable, replacing + * any previous setting. Truncate any trailing slash. + * @param repo is (URL) string indicating which repository contains this executable + * @param newpath is the path, relative to the repository, to the executable + * @throws IllegalArgumentException if invalid repo URL specified + */ + protected void setRepository(String repo, String newpath) { + repository = null; + if (repo != null) { + URL ghidraURL; + try { + ghidraURL = new URL(repo); + if (!GhidraURL.isGhidraURL(repo) || (!GhidraURL.isServerRepositoryURL(ghidraURL) && + !GhidraURL.isLocalProjectURL(ghidraURL))) { + throw new IllegalArgumentException("Unsupported repository URL: " + repo); + } + } + catch (MalformedURLException e) { + throw new IllegalArgumentException("Unsupported repository URL: " + repo, e); + } + URL projectURL = GhidraURL.getProjectURL(ghidraURL); + repository = projectURL.toExternalForm(); + } + + path = newpath; + if ((path != null) && (path.charAt(path.length() - 1) == '/')) { // No slash at end of path string + if (path.length() == 1) { + path = null; + } + else { + path = path.substring(0, path.length() - 1); + } + } + if ((path != null) && (path.charAt(0) == '/')) { // No slash at beginning of path + if (path.length() == 1) { + path = null; + } + else { + path = path.substring(1); + } + } + } + + /** + * Set the ingest date of the executable + * @param dt is the data, which may be null + */ + private void setDate(Date dt) { + if (dt == null) { + date = EMPTY_DATE; + } + else { + date = dt; + } + } + + protected void setRowId(RowKey i) { + rowid = i; + } + + protected void setAlreadyStored() { + flags |= ALREADY_STORED; + } + + protected void setXrefIndex(int val) { + xrefIndex = val; + } + + protected void setCategory(List cats) { + flags |= CATEGORIES_SET; + if (cats == null || cats.size() == 0) { + usercat = null; + return; + } + usercat = cats; + Collections.sort(usercat); // keep categories sorted, by type, then by category + } + + protected void cloneCategories(ExecutableRecord op2) { + flags &= ~CATEGORIES_SET; + if (op2.categoriesAreSet()) { + flags |= CATEGORIES_SET; + } + if (op2.usercat == null) { + return; + } + usercat = new ArrayList(); + for (int i = 0; i < op2.usercat.size(); ++i) { + CategoryRecord curRec = op2.usercat.get(i); + CategoryRecord cloneRec = new CategoryRecord(curRec.getType(), curRec.getCategory()); + usercat.add(cloneRec); + } + } + + /** + * @return the list of {@link CategoryRecord}s associated with this executable + */ + public List getAllCategories() { + return usercat; + } + + /** + * Return the executable's settings for a specific category type + * @param type is the category type + * @return the list of settings with this type (or null) + */ + public List getCategory(String type) { + if (usercat == null) { + return null; + } + List res = new ArrayList(); + int min = 0; + int max = usercat.size() - 1; + while (min <= max) { + int mid = (min + max) / 2; + String curtype = usercat.get(mid).getType(); + int cmp = type.compareTo(curtype); + if (cmp <= 0) { + max = mid - 1; + } + else { + min = mid + 1; + } + } + + while (min < usercat.size()) { + CategoryRecord currec = usercat.get(min); + if (!type.equals(currec.getType())) { + break; + } + min += 1; + res.add(currec.getCategory()); + } + return res; + } + + /** + * Determine if an executable has been set with a specific category value + * @param type is the type of category to check + * @param value is the value to check for + * @return true if the executable has that value, false otherwise + */ + public boolean hasCategory(String type, String value) { + if (usercat == null) { + return false; + } + int min = 0; + int max = usercat.size() - 1; + while (min <= max) { + int mid = (min + max) / 2; + CategoryRecord catrec = usercat.get(mid); + if (catrec == null) { + Msg.error(this, "No entry in category list found for index: " + mid + + " (list size = " + usercat.size() + ")"); + return false; + } + String curtype = catrec.getType(); + int cmp = type.compareTo(curtype); + if (cmp == 0) { + final int subcmp = value.compareTo(catrec.getCategory()); + if (subcmp < 0) { + max = mid - 1; + } + else if (subcmp > 0) { + min = mid + 1; + } + else { + return true; // Found match of type and value + } + } + else if (cmp < 0) { + max = mid - 1; + } + else { + min = mid + 1; + } + } + + return false; + } + + /** + * @return the MD5 hash of the executable + */ + public String getMd5() { + return md5sum; + } + + /** + * @return the name of the executable + */ + public String getNameExec() { + return executableName; + } + + /** + * @return the architecture associated with the executable + */ + public String getArchitecture() { + return architecture; + } + + /** + * @return the name of the compiler that built this executable + */ + public String getNameCompiler() { + return compilerName; + } + + /** + * @return the date this executable was ingested into the database + */ + public Date getDate() { + return date; + } + + /** + * @return the URL of the repository containing this executable + */ + public String getRepository() { + return repository; + } + + /** + * @return the (repository relative) path to the executable + */ + public String getPath() { + return path; + } + + /** + * @return true if this executable is a "library" (functions identified only by name) + */ + public boolean isLibrary() { + return ((flags & LIBRARY) != 0); + } + + /** + * @return true if this database record has already been stored in the database + */ + public boolean isAlreadyStored() { + return ((flags & ALREADY_STORED) != 0); + } + + /** + * @return true if categories have been queried in (does not mean that it has any categories) + */ + public boolean categoriesAreSet() { + return ((flags & CATEGORIES_SET) != 0); + } + + /** + * @return the fully formed URL to this executable or null + */ + public String getURLString() { + if (repository == null) { + return null; + } + final StringBuffer buf = new StringBuffer(); + buf.append(repository); + if (GhidraURL.isLocalGhidraURL(repository)) { + if (!repository.endsWith("?")) { + // local URLs add path as a query string + buf.append("?"); + } + } + if (path != null) { + buf.append('/').append(path); + } + buf.append('/').append(executableName); + return buf.toString(); + } + + /** + * Get all the category settings of a specific type in alphabetic order. + * Multiple values are returned in a single String separated by ',' + * @param type is the type of category to retrieve + * @return the concatenated list of settings + */ + public String getExeCategoryAlphabetic(String type) { + final List catrecs = getCategory(type); + if ((catrecs == null) || (catrecs.size() == 0)) { + return ""; + } + if (catrecs.size() == 1) { + return catrecs.get(0); + } + // If there are more than one categories they should already be sorted + final StringBuffer buf = new StringBuffer(); + for (int i = 0; i < 4; ++i) { // Spell out 4 at most + buf.append(catrecs.get(i)); + if (i + 1 >= catrecs.size()) { + break; + } + if (i < 3) { + buf.append(','); + } + } + return buf.toString(); + } + + /** + * @return the database (row) id of this executable object + */ + public RowKey getRowId() { + return rowid; + } + + /** + * @return the internal cross-referencing index for this executable + */ + public int getXrefIndex() { + return xrefIndex; + } + + /** + * Serialize this executable (meta-data) to an XML stream + * @param fwrite is the XML stream + * @throws IOException if there are I/O errors writing to the stream + */ + public void saveXml(Writer fwrite) throws IOException { + fwrite.append("\n"); + fwrite.append(" ").append(md5sum).append("\n"); + fwrite.append(" "); + SpecXmlUtils.xmlEscapeWriter(fwrite, executableName); + fwrite.append("\n"); + fwrite.append(" "); + SpecXmlUtils.xmlEscapeWriter(fwrite, architecture); + fwrite.append("\n"); + fwrite.append(" "); + SpecXmlUtils.xmlEscapeWriter(fwrite, compilerName); + fwrite.append("\n"); + long seconds = date.getTime(); + final long millis = seconds % 1000; + seconds /= 1000; + fwrite.append(" "); + fwrite.append(SpecXmlUtils.encodeUnsignedInteger(seconds)); + fwrite.append("\n"); + if (repository != null) { + fwrite.append(" "); + SpecXmlUtils.xmlEscapeWriter(fwrite, repository); + fwrite.append("\n"); + } + if (path != null) { + fwrite.append(" "); + SpecXmlUtils.xmlEscapeWriter(fwrite, path); + fwrite.append("\n"); + } + + if (usercat != null) { + for (CategoryRecord element : usercat) { + element.saveXml(fwrite); + } + } + fwrite.append("\n"); + } + + /** + * Identify whether an md5 string is a placeholder hash + * (as generated by {@link ExecutableRecord#calcLibraryMd5Placeholder}) + * @param md5 is the md5 string + * @return true if it is a placeholder, false otherwise + */ + public static boolean isLibraryHash(String md5) { + if (md5.length() != 32) { + return false; + } + return md5.startsWith("bbbbbbbbaaaaaaaa"); + } + + /** + * Build a new {@link ExecutableRecord} by deserializing from an XML stream + * @param parser is the XML parser + * @param man is the DescriptionManager that should hold the new executable + * @return the new ExecutableRecord + * @throws LSHException if there are inconsistencies in the XML description + */ + public static ExecutableRecord restoreXml(XmlPullParser parser, DescriptionManager man) + throws LSHException { + final XmlElement el = parser.start("exe"); + final boolean islib = SpecXmlUtils.decodeBoolean(el.getAttribute("library")); + parser.start("md5"); + final String md5sum = parser.end().getText(); + parser.start("name"); + final String name_exec = parser.end().getText(); + String name_compiler = ""; + String architecture = ""; + long seconds = 0; + long millis = 0; + RowKey id = null; + String repo = null; + String path = null; + List cats = null; + while (parser.peek().isStart()) { + if (parser.peek().getName().equals("category")) { + if (cats == null) { + cats = new ArrayList(); + } + final CategoryRecord newrec = CategoryRecord.restoreXml(parser); + cats.add(newrec); + } + else { + final XmlElement subel = parser.start(); + final String nm = subel.getName(); + if (nm.equals("arch")) { + architecture = parser.end().getText(); + } + else if (nm.equals("compiler")) { + name_compiler = parser.end().getText(); + } + else if (nm.equals("date")) { + millis = SpecXmlUtils.decodeLong(subel.getAttribute("millis")); + if ((millis < 0) || (millis > 1000)) { + millis = 0; + } + seconds = SpecXmlUtils.decodeLong(parser.end().getText()); + } + else if (nm.equals("repository")) { + repo = parser.end().getText(); + } + else if (nm.equals("path")) { + path = parser.end().getText(); + } + else { + parser.end(); + } + } + } + + parser.end(el); + ExecutableRecord res; + + if (islib) { + res = man.newExecutableLibrary(name_exec, architecture, id); + if ((!res.getMd5().equals(md5sum))) { + throw new LSHException("Read bad library placeholder md5 for ExecutableRecord"); + } + } + else { + final long date_milli = seconds * 1000 + millis; + res = man.newExecutableRecord(md5sum, name_exec, name_compiler, architecture, + new Date(date_milli), repo, path, id); + } + res.setCategory(cats); + return res; + } + + @Override + public boolean equals(Object obj) { + if (obj == null) { + return false; + } + + if (this == obj) { + return true; + } + + final ExecutableRecord o = (ExecutableRecord) obj; + return md5sum.equals(o.md5sum); + } + + @Override + public int hashCode() { + return md5sum.hashCode(); + } + + @Override + public String toString() { + // @formatter:off + return getClass().getSimpleName() + "\n\t" + + "Executable Name: " + executableName + "\n\t" + + "Architecture: " + architecture + "\n\t" + + "Compiler Name: " + compilerName + "\n\t" + + "Path: " + path; + + // @formatter:off + } + + @Override + public int compareTo(ExecutableRecord o) { + if (this == o) { + return 0; + } + int comp; + comp = md5sum.compareTo(o.md5sum); + return comp; + } + + /** + * Compare just the metadata portion (names and versions) of two ExecutableRecords + * We do NOT compare categories as these may not have been read into the object yet + * @param o is ExecutableRecord to compare with this + * @return bit vector with a 1 bit for every field that differs + */ + public int compareMetadata(ExecutableRecord o) { + int res = 0; + if (!executableName.equals(o.executableName)) { + res |= METADATA_NAME; + } + if (!architecture.equals(o.architecture)) { + res |= METADATA_ARCH; + } + if ((flags & LIBRARY) != (o.flags & LIBRARY)) { + res |= METADATA_LIBR; + } + if ((flags & LIBRARY)!=0) + { + return res; // If we are comparing libraries, remaining fields aren't compared + } + if (!compilerName.equals(o.compilerName)) { + res |= METADATA_COMP; + } + if (!date.equals(o.date)) { + res |= METADATA_DATE; + } + if (repository == null) { + if (o.repository != null) { + res |= METADATA_REPO; + } + } + else { + if (o.repository == null) { + res |= METADATA_REPO; + } + else if (!repository.equals(o.repository)) { + res |= METADATA_REPO; + } + } + if (path == null) { + if (o.path != null) { + res |= METADATA_PATH; + } + } + else { + if (o.path == null) { + res |= METADATA_PATH; + } + else if (!path.equals(o.path)) { + res |= METADATA_PATH; + } + } + return res; + } + + /** + * Compare the set of categories that -this- and -op2- belong to + * @param op2 is executable to compare with this + * @return true if the categories are exactly the same + */ + public boolean compareCategory(ExecutableRecord op2) { + if (usercat == null) { + if (op2.usercat == null) { + return true; + } + return false; + } + if (op2.usercat == null) { + return false; + } + if (usercat.size() != op2.usercat.size()) { + return false; + } + for(int i=0;i findInsertions(List oldlist, + List newlist) { + if (newlist == null && oldlist != null) + { + return null; // Indicate we need to delete + } + if (newlist != null && oldlist == null) { + return newlist; + } + final List insert = new ArrayList(); + int i=0; + int j=0; + while(i { + private final ExecutableRecord exerec; + private final String function_name; // Name of the function (unique within the executable) + private final long address; // Address offset of this function within its executable or -1 for a library function + private SignatureRecord sigrec; + private List callrec; + private RowKey id; // table id of this description + private long vectorid; // vectorid of signature associated with this function + private int flags; // 1-bit attributes of the function + + public static class Update { + public FunctionDescription update; + public boolean function_name; // Do we update the function name + public boolean flags; // Do we update the flags + } + + public FunctionDescription(ExecutableRecord ex, String name, long addr) { + exerec = ex; + function_name = name; + sigrec = null; + id = null; + vectorid = 0; + address = addr; + flags = 0; + callrec = null; + } + + void setId(RowKey i) { + id = i; + } + + void setVectorId(long i) { + vectorid = i; + } + + void setFlags(int fl) { + flags = fl; + } + + void insertCall(FunctionDescription fd, int lhash) { + if (callrec == null) { + callrec = new ArrayList(); + } + callrec.add(new CallgraphEntry(fd, lhash)); + } + + public void setSignatureRecord(SignatureRecord srec) { + sigrec = srec; + } + + public String getFunctionName() { + return function_name; + } + + public ExecutableRecord getExecutableRecord() { + return exerec; + } + + public SignatureRecord getSignatureRecord() { + return sigrec; + } + + public List getCallgraphRecord() { + return callrec; + } + + public RowKey getId() { + return id; + } + + public long getVectorId() { + return vectorid; + } + + public long getAddress() { + return address; + } + + public int getFlags() { + return flags; + } + + @Override + public boolean equals(Object obj) { + FunctionDescription o = (FunctionDescription) obj; + int comp = exerec.compareTo(o.exerec); + if (comp != 0) { + return false; + } + comp = function_name.compareTo(o.function_name); + if (comp != 0) { + return false; + } + comp = Long.compareUnsigned(address, o.address); + return (comp == 0); + } + + @Override + public int hashCode() { + int val = (int) (address >> 32) + exerec.hashCode(); + val *= 151; + val ^= function_name.hashCode(); + val *= 13; + val ^= (int) address; + return val; + } + + @Override + public String toString() { + return getClass().getSimpleName() + " " + function_name + " (" + exerec.getNameExec() + ")"; + } + + @Override + public int compareTo(FunctionDescription o) { + int comp = exerec.compareTo(o.exerec); + if (comp != 0) { + return comp; + } + comp = function_name.compareTo(o.function_name); + if (comp != 0) { + return comp; + } + comp = Long.compareUnsigned(address, o.address); + return comp; + } + + public void sortCallgraph() { + if ((callrec == null) || (callrec.size() < 2)) { + return; // Nothing to do + } + Collections.sort(callrec); + // dedup the list + int i = 1; + for (int j = 1; j < callrec.size(); ++j) { + FunctionDescription callrecj = callrec.get(j).getFunctionDescription(); + FunctionDescription callrecjm = callrec.get(j - 1).getFunctionDescription(); + if (callrecj != callrecjm) { // Compare as pointers + if (i != j) { + callrec.set(i, callrec.get(j)); + } + i += 1; + } + } + if (i != callrec.size()) { + for (int j = callrec.size() - 1; j >= i; --j) { + callrec.remove(j); + } + } + } + + public String printRaw() { + StringBuilder buf = new StringBuilder(); + buf.append(function_name); + buf.append(' '); + buf.append(exerec.printRaw()); + return buf.toString(); + } + + public void saveXml(Writer fwrite) throws IOException { + fwrite.append(" 0)) { + fwrite.append("\" sigdup=\""); + fwrite.append(SpecXmlUtils.encodeUnsignedInteger(sigrec.getCount())); + } + fwrite.append("\">\n"); + if (sigrec != null) { + sigrec.saveXml(fwrite); + } + if (callrec != null) { + for (CallgraphEntry element : callrec) { + element.saveXml(this, fwrite); + } + } + if (flags != 0) { + fwrite.append(""); + fwrite.append(SpecXmlUtils.encodeUnsignedInteger(flags)); + fwrite.append("\n"); + } + fwrite.append("\n"); + } + + /** + * Update the boolean fields in -res- to true, for every field in -this- that needs to be updated from -fromDB- + * @param res stores the boolean results for which fields to update + * @param fromDB is the metadata to compare with -this- to decided if updates are necessary + * @return true if one or more updates is necessary + */ + public boolean diffForUpdate(Update res, FunctionDescription fromDB) { + res.function_name = !function_name.equals(fromDB.function_name); + flags = (0xfffffff9 & flags) | (fromDB.flags & 6); // keep bits 1 and 2 of database flags + res.flags = (flags != fromDB.flags); + id = fromDB.id; + res.update = this; + return res.function_name || res.flags; + } + + static public FunctionDescription restoreXml(XmlPullParser parser, + LSHVectorFactory vectorFactory, DescriptionManager man, ExecutableRecord erec) + throws LSHException { + int count = 0; + XmlElement el = parser.start("fdesc"); + String fname = el.getAttribute("name"); + String addrString = el.getAttribute("addr"); + String sigdupstr = el.getAttribute("sigdup"); + long address = -1; // Default value if no attribute present + if (addrString != null) { + address = SpecXmlUtils.decodeLong(addrString); + } + if (sigdupstr != null) { + count = SpecXmlUtils.decodeInt(sigdupstr); + } + FunctionDescription fdesc = man.newFunctionDescription(fname, address, erec); + if (parser.peek().isStart()) { + if (parser.peek().getName().equals("lshcosine")) { + SignatureRecord.restoreXml(parser, vectorFactory, man, fdesc, count); + } + while (parser.peek().isStart()) { + String nm = parser.peek().getName(); + if (nm.equals("flags")) { + parser.start(); + int flags = SpecXmlUtils.decodeInt(parser.end().getText()); + fdesc.flags = flags; + } + else { // Assume it is a callgraph entry + CallgraphEntry.restoreXml(parser, man, fdesc); + } + } + } + parser.end(); + return fdesc; + } + + /** + * Create a map from addresses to functions + * @param iter is the list of functions to map + * @return the Map + */ + public static Map createAddressToFunctionMap( + Iterator iter) { + TreeMap addrmap = new TreeMap(); + while (iter.hasNext()) { + FunctionDescription func = iter.next(); + long addr = func.getAddress(); + if (addr == -1) { + continue; + } + addrmap.put(addr, func); + } + return addrmap; + } + + /** + * Match new functions to old functions via the address, test if there is an update between the two functions, + * generate an update record if there is, return the list of updates + * @param iter is the list of NEW functions + * @param addrMap is a map from address to OLD functions + * @param badList is a container for new functions that could not be mapped to old + * @return the list of Update records + */ + public static List generateUpdates(Iterator iter, + Map addrMap, List badList) { + List updateList = new ArrayList(); + Update curupdate = new Update(); + while (iter.hasNext()) { + FunctionDescription newfunc = iter.next(); + long addr = newfunc.getAddress(); + if (addr == -1) { + continue; + } + FunctionDescription oldfunc = addrMap.get(addr); + if (oldfunc == null) { + badList.add(newfunc); // Keep track of functions with update info which we couldn't find + continue; + } + if (newfunc.diffForUpdate(curupdate, oldfunc)) { // Check if there is any change in metadata + updateList.add(curupdate); + curupdate = new FunctionDescription.Update(); + } + } + return updateList; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/FunctionDescriptionMapper.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/FunctionDescriptionMapper.java new file mode 100755 index 0000000000..1cdf5082e7 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/FunctionDescriptionMapper.java @@ -0,0 +1,57 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +import java.io.IOException; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.xml.XmlPullParser; + +/** + * Scan a description XML file and for each tag, parse it, build the FunctionDescription + * object and call handleFunction + * + */ +public abstract class FunctionDescriptionMapper { + protected int recnum; // Index of current FunctionDescription being processed + + public abstract void handleExecutable(ExecutableRecord erec) throws IOException, InterruptedException; + + public abstract void handleFunction(FunctionDescription fdesc,int rnum) throws IOException, InterruptedException; + + public void processFile(XmlPullParser parser,LSHVectorFactory vectorFactory) throws IOException, + InterruptedException, LSHException { + recnum = 0; + + DescriptionManager dmanage = null; + parser.start("description"); + while(parser.peek().isStart()) { + parser.start("execlist"); + dmanage = new DescriptionManager(); // Allocate per executable + ExecutableRecord erec = ExecutableRecord.restoreXml(parser, dmanage); + handleExecutable(erec); + while(parser.peek().isStart()) { + FunctionDescription fdesc = FunctionDescription.restoreXml(parser, vectorFactory, dmanage, erec); + handleFunction(fdesc,recnum); + dmanage.clearFunctions(); // Free up memory + recnum += 1; // Count the record + } + parser.end(); + } + parser.end(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/RowKey.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/RowKey.java new file mode 100755 index 0000000000..9a121dd1cc --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/RowKey.java @@ -0,0 +1,24 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +public abstract class RowKey implements Comparable { + + /** + * @return the (least significant) 64-bits of the row key + */ + public abstract long getLong(); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/SignatureRecord.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/SignatureRecord.java new file mode 100755 index 0000000000..b36e1e3623 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/SignatureRecord.java @@ -0,0 +1,103 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.xml.XmlPullParser; + +import java.io.IOException; +import java.io.Writer; + +public class SignatureRecord { + private LSHVector sigvector; // Vector of signatures + private long vectorid; // vectorid of signature + private int count; // Number of duplicates of this signature within the database + + public SignatureRecord(LSHVector v) { + sigvector = v; + vectorid = 0; + count = 0; + } + + void setVectorId(long i) { + vectorid = i; + } + + void setCount(int c) { + count = c; + } + + public LSHVector getLSHVector() { + return sigvector; + } + + public long getVectorId() { + return vectorid; + } + + public int getCount() { + return count; + } + + public void saveXml(Writer fwrite) throws IOException { + sigvector.saveXml(fwrite); + } + + public static void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory, + DescriptionManager man, FunctionDescription fdesc, int count) { + SignatureRecord srec = man.newSignature(parser, vectorFactory, count); + man.attachSignature(fdesc, srec); + } + + /* (non-Javadoc) + * @see java.lang.Object#hashCode() + */ + @Override + public int hashCode() { + final int prime = 31; + int result = 1; + result = prime * result + ((sigvector == null) ? 0 : sigvector.hashCode()); + return result; + } + + /* (non-Javadoc) + * @see java.lang.Object#equals(java.lang.Object) + */ + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null) { + return false; + } + if (!(obj instanceof SignatureRecord)) { + return false; + } + SignatureRecord other = (SignatureRecord) obj; + if (sigvector == null) { + if (other.sigvector != null) { + return false; + } + } + else if (!sigvector.equals(other.sigvector)) { + return false; + } + return true; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/VectorResult.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/VectorResult.java new file mode 100755 index 0000000000..216ce9de41 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/description/VectorResult.java @@ -0,0 +1,100 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.description; + +import java.io.IOException; +import java.io.Writer; +import java.util.Objects; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +public class VectorResult { + public long vectorid; // Id of vector + public double sim; // Similarity score + public double signif; // Significance score + public int hitcount; // Number of duplicate results + public LSHVector vec; + + public VectorResult() { + vectorid = 0; + hitcount = 0; + vec = null; + } + + public VectorResult(long vid, int cnt, double sm, double sg, LSHVector v) { + vectorid = vid; + sim = sm; + signif = sg; + hitcount = cnt; + vec = v; + } + + public void saveXml(Writer write) throws IOException { + StringBuilder buf = new StringBuilder(); + buf.append("\n"); + buf.append(" ").append(hitcount).append("\n"); + buf.append(" ").append(Double.toString(sim)).append("\n"); + buf.append(" ").append(Double.toString(signif)).append("\n"); + write.append(buf.toString()); + if (vec != null) + vec.saveXml(write); + write.append("\n"); + } + + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) { + XmlElement el = parser.start("vec"); + vectorid = SpecXmlUtils.decodeLong(el.getAttribute("id")); + parser.start("hit"); + hitcount = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("sim"); + sim = Double.parseDouble(parser.end().getText()); + parser.start("sig"); + signif = Double.parseDouble(parser.end().getText()); + if (parser.peek().isStart()) { + vec = vectorFactory.restoreVectorFromXml(parser); + } + parser.end(); + } + + //generated by Eclipse + @Override + public int hashCode() { + return Objects.hash(hitcount, signif, sim, vec, vectorid); + } + + //generated by Eclipse + @Override + public boolean equals(Object obj) { + if (this == obj) + return true; + if (obj == null) + return false; + if (getClass() != obj.getClass()) + return false; + VectorResult other = (VectorResult) obj; + return hitcount == other.hitcount && + Double.doubleToLongBits(signif) == Double.doubleToLongBits(other.signif) && + Double.doubleToLongBits(sim) == Double.doubleToLongBits(other.sim) && + Objects.equals(vec, other.vec) && vectorid == other.vectorid; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/Base64Lite.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/Base64Lite.java new file mode 100755 index 0000000000..10d38152bf --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/Base64Lite.java @@ -0,0 +1,120 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +import java.util.Arrays; + +/** + * Lightweight Base64 encoder for writing chars directly to StringBuilders and giving + * direct access to the encode and decode arrays + * + */ +public class Base64Lite { + // URL and Filename safe alphabet, RFC 4648 Table 2 + public static final char[] encode = { + 'A', 'B', 'C', 'D', 'E', 'F', 'G', 'H', 'I', 'J', 'K', 'L', 'M', + 'N', 'O', 'P', 'Q', 'R', 'S', 'T', 'U', 'V', 'W', 'X', 'Y', 'Z', + 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', + 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z', + '0', '1', '2', '3', '4', '5', '6', '7', '8', '9', '-', '_' + }; + public static final int[] decode= new int[128]; + static { + Arrays.fill(decode, -1); // -1 for any invalid character position + for (int i = 0; i < encode.length; i++) + decode[encode[i]] = i; + } + + /** + * Encode a long value in base64 to a StringBuilder "stream". + * Omit initial 'A' characters if the high-order bits of the value are zero + * @param buf is the buffer to write to + * @param val is the long value to encode + */ + public static void encodeLongBase64(StringBuilder buf,long val) { + boolean seenNonZero = false; + if (val == 0) { + buf.append(encode[0]); + return; + } + for(int i=60;i>=0;i-=6) { + int chunk = (int)(val >> i) & 0x3f; + if (chunk == 0 && seenNonZero) { + buf.append(encode[chunk]); + } + else { + buf.append(encode[chunk]); + seenNonZero = true; + } + } + } + + /** + * Encode a long value in base64 to the StringBuilder "stream" padding out with 'A' characters + * so that exactly 11 characters are always written to the stream + * @param buf is the buffer to write to + * @param val is the long value to encode + */ + public static void encodeLongBase64Padded(StringBuilder buf,long val) { + for(int i=60;i>=0;i-=6) { + int chunk = (int)(val >> i) & 0x3f; + buf.append(encode[chunk]); + } + } + + /** + * Encode a long value in base64 to a String. Omit initial 'A' characters if the high-order bits of the value are zero + * @param val is the long to encode + * @return the encoded String + */ + public static String encodeLongBase64(long val) { + char[] buffer = new char[11]; + if (val == 0) { + buffer[0] = encode[0]; + return new String(buffer,0,1); + } + int pos = 0; + boolean seenNonZero = false; + for(int i=60;i>=0;i-=6) { + int chunk = (int)(val >> i) & 0x3f; + if (chunk == 0 && seenNonZero) { + buffer[pos++] = encode[chunk]; + } + else { + buffer[pos++] = encode[chunk]; + seenNonZero = true; + } + } + return new String(buffer,0,pos); + } + + /** + * Decode (up to 11) base64 characters to produce a long + * @param val is the String to decode + * @return the decode long + */ + public static long decodeLongBase64(String val) { + long res = 0; + for(int i=0;i= 200) && (lastResponseCode < 300); + } + + /** + * Start a new request to the elastic server. This establishes the OutputStream for writing the body of the request + * @param command is the type of command + * @param path is the overarching index/type/ path + * @throws IOException for problems with the socket + */ + public void startHttpRequest(String command, String path) throws IOException { + URL httpURL = new URL(httpURLbase + path); + connection = (HttpURLConnection) httpURL.openConnection(); + connection.setRequestMethod(command); + connection.setRequestProperty("Content-Type", "application/json"); + connection.setDoOutput(true); + writer = new OutputStreamWriter(connection.getOutputStream()); + } + + public void startHttpBulkRequest(String bulkCommand) throws IOException { + URL httpURL = new URL(hostURL + bulkCommand); + connection = (HttpURLConnection) httpURL.openConnection(); + connection.setRequestMethod(POST); + connection.setRequestProperty("Content-Type", "application/x-ndjson"); + connection.setDoOutput(true); + writer = new OutputStreamWriter(connection.getOutputStream()); + } + + public void startHttpRawRequest(String command, String path) throws IOException { + URL httpURL = new URL(hostURL + path); + connection = (HttpURLConnection) httpURL.openConnection(); + connection.setRequestMethod(command); + connection.setRequestProperty("Content-Type", "application/json"); + connection.setDoOutput(true); + writer = new OutputStreamWriter(connection.getOutputStream()); + } + + /** + * Start a request with no input body, URI only + * @param command is the command to issue + * @param path is the overarching request path: index/... + * @throws IOException for problems with the socket + */ + public void startHttpURICommand(String command, String path) throws IOException { + URL httpURL = new URL(httpURLbase + path); + connection = (HttpURLConnection) httpURL.openConnection(); + connection.setRequestMethod(command); + connection.setDoOutput(true); + } + + /** + * Assuming the writer has been closed and connection.getResponseCode() is called + * placing the value in lastResponseCode, read the response and parse into a JSONObject + * @return the JSONObject + * @throws IOException for problems with the socket + * @throws ParseException for JSON parse errors + */ + private JSONObject grabResponse() throws IOException, ParseException { + JSONParser parser = new JSONParser(); + Reader reader; + if (lastRequestSuccessful()) { + reader = new InputStreamReader(connection.getInputStream()); + } + else { + reader = new InputStreamReader(connection.getErrorStream()); + } + JSONObject jsonObject = (JSONObject) parser.parse(reader); + return jsonObject; + } + + /** + * Elastic search sends a JSON document in the Http error stream for any error + * Pull out relevant info from the document and construct an exception message + * @param resp is the parsed error document + * @return the exception String + */ + private String parseErrorJSON(JSONObject resp) { + Object errorObj = resp.get("error"); + if (errorObj == null) { + return "Unknown error format"; + } + if (errorObj instanceof String) { + return (String) errorObj; + } + if (!(errorObj instanceof JSONObject)) { + return "Unknown error format"; + } + JSONObject jsonObj = (JSONObject) errorObj; + String typeString = (String) jsonObj.get("type"); + String reasonString = (String) jsonObj.get("reason"); + if (typeString == null) { + typeString = "Unknown Error"; + } + if (reasonString == null) { + reasonString = "Unknown reason"; + } + return typeString + " : " + reasonString; + } + + /** + * Send a raw request to the server that is not specific to the repository. + * Intended for general configuration or security commands + * @param command is the type of command + * @param path is the specific URL path receiving the command + * @param body is JSON document describing the command + * @return the response as parsed JSONObject + * @throws ElasticException for any problems with the connection + */ + public JSONObject executeRawStatement(String command, String path, String body) + throws ElasticException { + try { + startHttpRawRequest(command, path); + writer.write(body); + writer.close(); + lastResponseCode = connection.getResponseCode(); + JSONObject resp = grabResponse(); + if (!lastRequestSuccessful()) { + throw new ElasticException(parseErrorJSON(resp)); + } + return resp; + } + catch (IOException e) { + throw new ElasticException("Error sending request: " + e.getMessage()); + } + catch (ParseException e) { + throw new ElasticException("Error parsing response: " + e.getMessage()); + } + + } + + /** + * Execute an elasticsearch command where we are not expecting a response + * @param command is the type of the command + * @param path is the overarching index/type/ + * @param body is the JSON document describing the request + * @throws ElasticException for any problems with the connecting + */ + public void executeStatementNoResponse(String command, String path, String body) + throws ElasticException { + try { + startHttpRequest(command, path); + writer.write(body); + writer.close(); + lastResponseCode = connection.getResponseCode(); + JSONObject resp = grabResponse(); + if (!lastRequestSuccessful()) { + throw new ElasticException(parseErrorJSON(resp)); + } + } + catch (IOException e) { + throw new ElasticException("Error sending request: " + e.getMessage()); + } + catch (ParseException e) { + throw new ElasticException("Error parsing response: " + e.getMessage()); + } + } + + /** + * Execute an elastic search statement and return the JSON response to user + * @param command is the type of command + * @param path is the overarching index/type/ + * @param body is JSON document describing the request + * @return the parsed response as a JSONObject + * @throws ElasticException for any problems with the connection + */ + public JSONObject executeStatement(String command, String path, String body) + throws ElasticException { + try { + startHttpRequest(command, path); + writer.write(body); + writer.close(); + lastResponseCode = connection.getResponseCode(); + JSONObject resp = grabResponse(); + if (!lastRequestSuccessful()) { + throw new ElasticException(parseErrorJSON(resp)); + } + return resp; + } + catch (IOException e) { + throw new ElasticException("Error sending request: " + e.getMessage()); + } + catch (ParseException e) { + throw new ElasticException("Error parsing response: " + e.getMessage()); + } + } + + /** + * Execute an elastic search statement and return the JSON response to user + * Do not throw an exception on failure, just return the error response + * @param command is the type of command + * @param path is the overarching index/type/ + * @param body is JSON document describing the request + * @return the parsed response as a JSONObject + * @throws ElasticException for any problems with the connection + */ + public JSONObject executeStatementExpectFailure(String command, String path, String body) + throws ElasticException { + try { + startHttpRequest(command, path); + writer.write(body); + writer.close(); + lastResponseCode = connection.getResponseCode(); + JSONObject resp = grabResponse(); + return resp; + } + catch (IOException e) { + throw new ElasticException("Error sending request: " + e.getMessage()); + } + catch (ParseException e) { + throw new ElasticException("Error parsing response: " + e.getMessage()); + } + } + + /** + * Send a bulk request to the elasticsearch server. This is a special format for combining multiple commands + * and is structured slightly differently from other commands. + * @param path is the specific URL path receiving the bulk command + * @param body is structured list of JSON commands and source + * @return the response as parsed JSONObject + * @throws ElasticException for any problems with the connection + */ + public JSONObject executeBulk(String path, String body) throws ElasticException { + try { + startHttpBulkRequest(path); + writer.write(body); + writer.close(); + lastResponseCode = connection.getResponseCode(); + JSONObject resp = grabResponse(); + if (!lastRequestSuccessful()) { + throw new ElasticException(parseErrorJSON(resp)); + } + return resp; + } + catch (IOException e) { + throw new ElasticException("Error sending request: " + e.getMessage()); + } + catch (ParseException e) { + throw new ElasticException("Error parsing response: " + e.getMessage()); + } + } + + public JSONObject executeURIOnly(String command, String path) throws ElasticException { + try { + startHttpURICommand(command, path); + lastResponseCode = connection.getResponseCode(); + JSONObject resp = grabResponse(); + if (!lastRequestSuccessful()) { + throw new ElasticException(parseErrorJSON(resp)); + } + return resp; + } + catch (IOException e) { + throw new ElasticException("Error sending request: " + e.getMessage()); + } + catch (ParseException e) { + throw new ElasticException("Error parsing response: " + e.getMessage()); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java new file mode 100755 index 0000000000..2d49fb8fb1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticDatabase.java @@ -0,0 +1,3493 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +import java.io.IOException; +import java.io.StringReader; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.*; +import java.util.Map.Entry; + +import org.apache.commons.lang3.StringUtils; +import org.json.simple.JSONArray; +import org.json.simple.JSONObject; + +import generic.lsh.vector.*; +import ghidra.features.bsim.gui.filters.FunctionTagBSimFilterType; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.client.*; +import ghidra.features.bsim.query.client.tables.ExeTable.ExeTableOrderColumn; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.*; +import ghidra.framework.client.ClientUtil; +import ghidra.util.xml.SpecXmlUtils; + +/** + * Implement the BSim database interface on top of an ElasticSearch back-end + * ElasticSearch holds records as JSON documents. Documents + * are stored in a specific "index". The primary BSim document index/types are: + * executable/exe is executable metadata corresponding to the ExecutableRecord object + * executable/function is function metadata corresponding to the FunctionDescription object + * vector/vector is the main feature vector corresponding to an LSHVector object + * meta/meta is a document containing the duplication count for a particular feature vector + */ +public class ElasticDatabase implements FunctionDatabase { + + public static final int LAYOUT_VERSION = 3; // Version of the BSim schema within ElasticSearch as implemented + // by this database implementation, so clients can detect + // incompatible servers + public static final int MAX_VECTOR_OVERALL = 9000; // Max vectors return in one query (Note: index.max_result_window defaults to 10000) + public static final int MAX_FUNCTION_WINDOW = 500; // Max functions returned per window, when querying single executable + public static final int MAX_FUNCTIONUPDATE_WINDOW = 500; // Max functions updated in one window + public static final int MAX_VECTORCOUNT_WINDOW = 100; // Max vector meta documents returned in one mget + public static final int MAX_VECTORDELETE_WINDOW = 100; // Max vector or meta documents to delete/update in one bulk request + public static final int MAX_FUNCTION_BULK = 200; // Maximum functions ingested in one bulk request + public static final int MAX_VECTOR_BULK = 200; // Maximum vectors ingested in one bulk request + + private ElasticConnection connection; // Low-level connection to the database + private String userName = null; // User name for server authentication + private ConnectionType connectionType = ConnectionType.Unencrypted_No_Authentication; + private DatabaseInformation info; // Information about the active database + private Base64VectorFactory vectorFactory; // factory used to create BSim feature vectors + private final BSimServerInfo serverInfo; // NOTE: does not reflect the use of http vs https + private final String baseURL; // Base URL for connecting to elasticsearch, i.e. http://hostname:9200 + private final String repository; // Name of the repository, prefix to all elasticsearch indices + private Error lastError; // Info on error caused by last action taken on this interface (null if no error) + private Status status; // status of the connection + private boolean initialized; // true if the connection has been successfully initialized + + /** + * Append a list of CategoryRecords as (part of) a JSON document to a StringBuilder + * Used as part of constructing JSON serialization of ExecutableRecords + * @param catRecords list of category records + * @param buffer for writing + */ + private static void appendCategoryTag(List catRecords, StringBuilder buffer) { + buffer.append("\"execategory\": ["); + if (catRecords != null) { + boolean needsComma = false; + for (CategoryRecord catrec : catRecords) { + if (needsComma) { + buffer.append(','); + } + else { + needsComma = true; + } + buffer.append('\"'); + // Append type/value pair as concatendated strings separated by a TAB. + // When sorting as strings, this should give the same order as sorted CategoryRecords + buffer.append(catrec.getType()) + .append("\\t") + .append(JSONObject.escape(catrec.getCategory())); + buffer.append('\"'); + } + } + buffer.append(']'); + } + + /** + * Write an ExecutableRecord (meta-data about an executable) to the "executable" index. + * @param exeRecord is the record to write + * @param exeId is the unique id for the elasticsearch document + * @return true if the document gets created, return false is the document already exists + * @throws ElasticException if there are problems communicating with the server + */ + private boolean insertExecutableRecord(ExecutableRecord exeRecord, String exeId) + throws ElasticException { + StringBuilder builder = new StringBuilder(); + builder.append("{ \"md5\": \"").append(exeRecord.getMd5()).append("\", "); + builder.append("\"name_exec\": \"") + .append(JSONObject.escape(exeRecord.getNameExec())) + .append("\", "); + builder.append("\"architecture\": \"").append(exeRecord.getArchitecture()).append("\", "); + builder.append("\"name_compiler\": \"").append(exeRecord.getNameCompiler()).append("\", "); + builder.append("\"ingest_date\": ").append(exeRecord.getDate().getTime()).append(", "); + if (exeRecord.getRepository() == null) { + builder.append("\"repository\": null, "); + } + else { + builder.append("\"repository\": \"") + .append(JSONObject.escape(exeRecord.getRepository())) + .append("\", "); + } + if (exeRecord.getPath() == null) { + builder.append("\"path\": null"); + } + else { + builder.append("\"path\": \"") + .append(JSONObject.escape(exeRecord.getPath())) + .append("\""); + } + List catrecs = exeRecord.getAllCategories(); + if (catrecs != null) { + builder.append(", "); + appendCategoryTag(catrecs, builder); + } + builder.append(", \"join_field\": \"exe\" }"); + StringBuilder pathbuilder = new StringBuilder(); + pathbuilder.append("executable/_doc/"); + pathbuilder.append(exeId); + pathbuilder.append("?op_type=create"); // Do "create" operation, so we fail if document already exists + JSONObject resp = connection.executeStatementExpectFailure(ElasticConnection.PUT, + pathbuilder.toString(), builder.toString()); + JSONObject error = (JSONObject) resp.get("error"); + if (error != null) { + String type = (String) error.get("type"); + if (type.startsWith("version_conflict")) { + return false; // Document already inserted + } + String reason = (String) error.get("reason"); + throw new ElasticException(reason); + } + return true; + } + + /** + * Set the "document id" for an ExecutableRecord. This is currently the + * last 96-bits of the md5 hash of the executable encoded in base64 + * @param manager is the container for the ExecutableRecord + * @param exeRecord has its key set + * @return the new RowKey + */ + private static RowKeyElastic updateKey(DescriptionManager manager, ExecutableRecord exeRecord) { + if (exeRecord.getRowId() == null) { + RowKeyElastic eKey = new RowKeyElastic(exeRecord.getMd5()); + manager.setExeRowId(exeRecord, eKey); + return eKey; + } + return (RowKeyElastic) exeRecord.getRowId(); + } + + /** + * Generate a sorted list of the document ids of the children of a function + * @param manager is the container of the function + * @param funcRecord is the description of the function + * @return the list of document ids as Strings + */ + private static List generateChildIds(DescriptionManager manager, + FunctionDescription funcRecord) { + List callgraphRecord = funcRecord.getCallgraphRecord(); + if (callgraphRecord == null) { + return null; + } + List res = new ArrayList<>(callgraphRecord.size()); + for (CallgraphEntry element : callgraphRecord) { + FunctionDescription func = element.getFunctionDescription(); + ExecutableRecord exeRec = func.getExecutableRecord(); + RowKeyElastic eKey = updateKey(manager, exeRec); + StringBuilder buffer = new StringBuilder(); + eKey.generateFunctionId(buffer, func); + res.add(buffer.toString()); + } + Collections.sort(res); + return res; + } + + /** + * Insert a range of FunctionDescription documents into the index. + * Documents are stored in the "executable" index + * @param manager is the container of the FunctionDescription objects + * @param exeRecord is the single executable containing the functions + * @param exeKey is the precomputed key of the executable + * @param exeId is the executable key encoded as a document id String + * @param iter is an iterator to the FunctionDescriptions to insert + * @param maxNumber is the (maximum) number of FunctionDescriptions to insert + * @throws ElasticException if there are problems communicating with the server + */ + private void insertFunctionRange(DescriptionManager manager, ExecutableRecord exeRecord, + RowKeyElastic exeKey, String exeId, Iterator iter, int maxNumber) + throws ElasticException { + StringBuilder builder = new StringBuilder(); + do { + FunctionDescription desc = iter.next(); + builder.append("{ \"create\": { \"_index\": \"") + .append(repository) + .append("_executable\", "); + builder.append("\"_id\": \""); + exeKey.generateFunctionId(builder, desc); + builder.append("\", \"routing\": \""); + builder.append(exeId).append("\"}}\n"); + builder.append("{ \"name_func\": \""); + builder.append(JSONObject.escape(desc.getFunctionName())); + SignatureRecord sigRec = desc.getSignatureRecord(); + long vecid = 0; + if (sigRec != null) { + vecid = sigRec.getVectorId(); + } + builder.append("\", \"id_signature\": \""); + Base64Lite.encodeLongBase64(builder, vecid); + builder.append("\", \"flags\": ").append(desc.getFlags()); + builder.append(", \"addr\": ").append(desc.getAddress()); + if (info.trackcallgraph) { + List vals = generateChildIds(manager, desc); + if (vals != null) { + builder.append(", \"childid\": ["); + boolean needComma = false; + for (String val : vals) { + if (needComma) { + builder.append(','); + } + builder.append('\"').append(val).append('\"'); + needComma = true; + } + builder.append(" ]"); + } + } + builder.append(", \"join_field\": { "); + builder.append("\"name\": \"function\", "); + builder.append("\"parent\": \""); + builder.append(exeId).append("\"}}\n"); + maxNumber -= 1; + if (maxNumber <= 0) { + break; + } + } + while (iter.hasNext()); + connection.executeBulk("/_bulk", builder.toString()); + } + + /** + * Query for a single executable document based on the md5. There should be 0 or 1 matching docs. + * @param md5 is the md5 string + * @return null or the "hit" portion of the response corresponding to the matching document + * @throws ElasticException for communication problems with the server + */ + private JSONObject queryMd5ExeMatch(String md5) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append( + "{ \"size\": 1, \"query\": { \"bool\": { \"filter\": { \"term\": { \"md5\": \""); + buffer.append(md5).append("\" } } } } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + JSONObject hits = (JSONObject) resp.get("hits"); + JSONObject totalRec = (JSONObject) hits.get("total"); + long total = (Long) totalRec.get("value"); + if (total == 0) { + return null; + } + JSONArray hitsArray = (JSONArray) hits.get("hits"); + return (JSONObject) hitsArray.get(0); + } + + /** + * Query for function documents matching a given executable and a given function name + * @param exeId is the document id of the executable to match + * @param functionName is the name of the function + * @param maxDocuments is the maximum number of documents to return + * @return a list of JSON function documents + * @throws ElasticException for communication problems with the server + */ + private JSONArray queryFuncNameMatch(String exeId, String functionName, int maxDocuments) + throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"size\": ").append(maxDocuments); + buffer.append(", \"_source\": { \"excludes\": [ \"childid\" ] }"); + buffer.append(", \"query\": {"); + buffer.append(" \"bool\": {"); + buffer.append(" \"must\": {"); + buffer.append(" \"term\": {"); + buffer.append(" \"name_func\": \""); + buffer.append(JSONObject.escape(functionName)); + buffer.append("\"} },"); + buffer.append(" \"filter\": {"); + buffer.append(" \"parent_id\": {"); + buffer.append(" \"type\": \"function\","); + buffer.append(" \"id\": \"").append(exeId); + buffer.append("\"} } } } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + JSONObject baseHits = (JSONObject) resp.get("hits"); + Object hitsArray = baseHits.get("hits"); + if (hitsArray == null) { + return new JSONArray(); + } + return (JSONArray) hitsArray; + } + + /** + * Query for function documents matching a given executable, + * a given function name, and a given function address. These 3 things + * should always identify a function uniquely within the database, so at + * most 1 document should be returned + * @param exeId is the document id of the executable to match + * @param functionName is the name of the function to match + * @param address is the address of the function to match + * @return the JSON function document or null if none match + * @throws ElasticException for communication problems with the server + */ + private JSONObject queryFuncNameAddress(String exeId, String functionName, long address) + throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"_source\": { \"excludes\": [ \"childid\" ] }"); + buffer.append(", \"query\": {"); + buffer.append(" \"bool\": {"); + buffer.append(" \"must\": {"); + buffer.append(" \"term\": {"); + buffer.append(" \"name_func\": \""); + buffer.append(JSONObject.escape(functionName)); + buffer.append("\"},"); + buffer.append(" \"term\": {"); + buffer.append(" \"addr\": ").append(address); + buffer.append("} },"); + buffer.append(" \"filter\": {"); + buffer.append(" \"parent_id\": {"); + buffer.append(" \"type\": \"function\","); + buffer.append(" \"id\": \"").append(exeId); + buffer.append("\"} } } } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + JSONObject baseHits = (JSONObject) resp.get("hits"); + JSONObject totalRec = (JSONObject) baseHits.get("total"); + long total = (Long) totalRec.get("value"); + if (total != 1) { + return null; + } + JSONArray hitsArray = (JSONArray) baseHits.get("hits"); + return (JSONObject) hitsArray.get(0); + } + + /** + * Query for a single executable given uniquely specifying information (ExeSpecifier). + * The exe document is retrieved from the database and parsed into an ExecutableRecord object. + * @param specifier is the uniquely specifying info about the executable + * @param manager is container for the final ExecutableRecord + * @return the matching ExecutableRecord or null if none is found + * @throws LSHException if there are problems adding the queried record to the container + * @throws ElasticException for communication problems with the server + */ + private ExecutableRecord findSingleExecutable(ExeSpecifier specifier, + DescriptionManager manager) throws LSHException, ElasticException { + if (specifier.exemd5 != null && specifier.exemd5.length() != 0) { + JSONObject row = queryMd5ExeMatch(specifier.exemd5); + if (row == null) { + return null; + } + return makeExecutableRecord(manager, row); + } + if (StringUtils.isEmpty(specifier.exename)) { + throw new LSHException("ExeSpecifier must provide either md5 or name"); + } + return querySingleExecutable(manager, specifier.exename, specifier.arch, + specifier.execompname); + } + + /** + * Query for a single executable given uniquely specifying information (ExeSpecifier). + * A cache map is checked first for a previously recovered ExecutableRecord object. + * If not in the cache, the database is searched. If the executable is found, the ExecutableRecord + * is parsed from the database document, put into the cache, and returned to the user. + * @param specifier is the uniquely specifying info about the executable + * @param manager is container for the final ExecutableRecord + * @param nameMap is the cache of previously recovered records + * @return the matching ExecutableRecord or null if none is found + * @throws LSHException if there are problems adding the queried record to the container + * @throws ElasticException for communication problems with the server + */ + private ExecutableRecord findSingleExeWithMap(ExeSpecifier specifier, + DescriptionManager manager, TreeMap nameMap) + throws LSHException, ElasticException { + ExecutableRecord erec = nameMap.get(specifier); + if (erec != null) { + return erec; + } + erec = findSingleExecutable(specifier, manager); + nameMap.put(specifier, erec); // Cache ExecutableRecord in map, even if its null + return erec; + } + + /** + * Within the list of all executables sorted by md5 or name, query for executables + * from a specific window in this list. + * @param manager is the container to receive the ExecutableRecords + * @param exeList will have recovered ExecutableRecords appended to the end + * @param maxWindow is (maximum) number of executables in the window + * @param searchAfter is the (md5 or name) of the last executable before the window + * or null if the first window is desired + * @param md5Order is true if executables are ordered by md5, false if ordered by name + * @param filter is an (optional) filter to apply to the list before constructing the window + * @throws ElasticException if there are errors querying the database + * @throws LSHException for problems creating the result set + */ + private void queryExecutables(DescriptionManager manager, List exeList, + int maxWindow, String searchAfter, boolean md5Order, String filter) + throws ElasticException, LSHException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"size\": ").append(maxWindow); + buffer.append(", \"query\": {"); + buffer.append(" \"bool\": {"); + buffer.append(" \"must\": {"); + buffer.append(" \"exists\": { \"field\": \"md5\" } }"); + if (filter != null) { + buffer.append(", "); + buffer.append(filter); + } + buffer.append("}}, "); + if (searchAfter != null) { + buffer.append("\"search_after\": [ \""); + buffer.append(JSONObject.escape(searchAfter)); + buffer.append("\"], "); + } + if (md5Order) { + buffer.append("\"sort\": [ { \"md5\": \"asc\" } ] }"); + } + else { + buffer.append("\"sort\": [ { \"name_exec\": \"asc\" } ] }"); + } + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + JSONObject baseHits = (JSONObject) resp.get("hits"); + JSONArray hitsArray = (JSONArray) baseHits.get("hits"); + for (Object element : hitsArray) { + JSONObject exerow = (JSONObject) element; + ExecutableRecord exeRecord = makeExecutableRecord(manager, exerow); + exeList.add(exeRecord); + } + } + + /** + * Place the same query for executables as {@link #queryExecutables(DescriptionManager, List, int, String, boolean, String)}. + * Except we only return the count of matching records. + * @param filter is the option filter options for the count + * @return the number of matching executables matching the filter + * @throws ElasticException is there is a server-side issue with the query + */ + protected int countExecutables(String filter) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"query\": {"); + buffer.append(" \"bool\": {"); + buffer.append(" \"must\": {"); + buffer.append(" \"exists\": { \"field\": \"md5\" } }"); + if (filter != null) { + buffer.append(", "); + buffer.append(filter); + } + buffer.append("}}}"); + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_count", + buffer.toString()); + Long res = (Long) resp.get("count"); + return res.intValue(); + } + + /** + * Query for a unique executable based on its name and possibly other metadata + * + * @param manager is the container to store the result + * @param exeName is the name the executable must match + * @param arch is the architecture the executable must match (may be zero length) + * @param compilerName is the compiler name the executable must match (may be zero length) + * @return the unique resulting ExecutableRecord or null, if none or more than 1 is found + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding new records to the container + */ + private ExecutableRecord querySingleExecutable(DescriptionManager manager, String exeName, + String arch, String compilerName) throws ElasticException, LSHException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"size\": 4,"); + buffer.append(" \"query\": {"); + buffer.append(" \"bool\": {"); + buffer.append(" \"must\": {"); + buffer.append(" \"term\": { \"name_exec\": \"") + .append(JSONObject.escape(exeName)) + .append("\" } }"); + if (!StringUtils.isEmpty(arch) || !StringUtils.isEmpty(compilerName)) { + buffer.append(", \"filter\": {"); + buffer.append(" \"script\": {"); + buffer.append(" \"script\": {"); + buffer.append(" \"inline\": \""); + if (StringUtils.isEmpty(arch)) { // cname only + buffer.append("doc['name_compiler'].value == params.comp"); + } + else if (StringUtils.isEmpty(compilerName)) { // arch only + buffer.append("doc['architecture'].value == params.arch"); + } + else { // Both are provided + buffer.append( + "doc['name_compiler'].value == params.comp && doc['architecture'].value == params.arch"); + } + buffer.append("\","); + buffer.append(" \"params\": {"); + if (arch.length() != 0) { + buffer.append(" \"arch\": \"").append(arch); + if (!StringUtils.isEmpty(compilerName)) { + buffer.append("\", "); + } + else { + buffer.append("\" "); + } + } + if (!StringUtils.isEmpty(compilerName)) { + buffer.append(" \"comp\": \"").append(compilerName).append("\" "); + } + buffer.append("}}}}"); + } + buffer.append("} } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + JSONObject baseHits = (JSONObject) resp.get("hits"); + JSONObject totalRec = (JSONObject) baseHits.get("total"); + long total = (Long) totalRec.get("value"); + if (total != 1) { + return null; // Either no results, or not unique + } + JSONArray hitsArray = (JSONArray) baseHits.get("hits"); + JSONObject exerow = (JSONObject) hitsArray.get(0); + ExecutableRecord exerec = makeExecutableRecord(manager, exerow); + return exerec; + } + + /** + * Fill in hitcounts for a list of VectorResults by querying for the meta document that + * matches the vector id. The meta documents are queried in bulk up to a maximum number. + * Two iterators pointing to the same list of VectorResults must be supplied. One is + * used to generate the bulk query document. The second is used to fill in the hitcount + * field from the resulting meta documents. If no exception is thrown, both iterators + * will be advanced the same number of times. + * @param iter1 is the iterator to VectorResults to fill in + * @param iter2 is a copy of the first iterator + * @param maxDocuments is the maximum number of documents to query for + * @return the total number of function matching the vectors queried + * @throws ElasticException for communication problems with the server + */ + private long fetchVectorCounts(Iterator iter1, Iterator iter2, + int maxDocuments) throws ElasticException { + if (!iter1.hasNext()) { + return 0; + } + long totalCount = 0; + VectorResult vecRes = iter1.next(); + StringBuilder buffer = new StringBuilder(); + // append the first id + buffer.append("{ \"ids\": [ "); + buffer.append('\"'); + Base64Lite.encodeLongBase64(buffer, vecRes.vectorid); + buffer.append('\"'); + for (int i = 1; i < maxDocuments; ++i) { + if (!iter1.hasNext()) { + break; + } + vecRes = iter1.next(); + buffer.append(", \""); + Base64Lite.encodeLongBase64(buffer, vecRes.vectorid); + buffer.append('\"'); + } + buffer.append(" ] }"); + JSONObject resp = + connection.executeStatement(ElasticConnection.GET, "meta/_mget", buffer.toString()); + JSONArray docs = (JSONArray) resp.get("docs"); + for (int i = 0; i < maxDocuments; ++i) { + if (!iter2.hasNext()) { + break; + } + vecRes = iter2.next(); + JSONObject oneResp = (JSONObject) docs.get(i); + String matchId = (String) oneResp.get("_id"); + long matchIdVal = Base64Lite.decodeLongBase64(matchId); + JSONObject source = (JSONObject) oneResp.get("_source"); + if (source == null) { + throw new ElasticException("meta document does not exist for id=" + matchId); + } + if (matchIdVal != vecRes.vectorid) { + throw new ElasticException("Mismatch in metaid"); + } + long count = (Long) source.get("count"); + totalCount += count; + vecRes.hitcount = (int) count; + } + return totalCount; + } + + /** + * Fetch vectors in bulk from the database, given a list of VectorResults with the vector ids + * The vector documents are queried, then the resulting LSHVector objects are filled + * in for the VectorResults by parsing the documents. Two iterators pointing to the same list + * of VectorResults are required, one for building the query, one for filling in the LSHVectors. + * If no exception is thrown, both iterators are advanced the same number of times. + * @param iter1 is the iterator to VectorResults to fill in + * @param iter2 is a copy of the first iterator + * @param maxDocuments is the maximum number of documents to query for + * @throws ElasticException for communication problems with the server + */ + private void fetchVectors(Iterator iter1, Iterator iter2, + int maxDocuments) throws ElasticException { + if (!iter1.hasNext()) { + return; + } + VectorResult vecRes = iter1.next(); + StringBuilder buffer = new StringBuilder(); + // append the first id + buffer.append("{ \"ids\": [ "); + buffer.append('\"'); + Base64Lite.encodeLongBase64(buffer, vecRes.vectorid); + buffer.append('\"'); + for (int i = 1; i < maxDocuments; ++i) { + if (!iter1.hasNext()) { + break; + } + vecRes = iter1.next(); + buffer.append(", \""); + Base64Lite.encodeLongBase64(buffer, vecRes.vectorid); + buffer.append('\"'); + } + buffer.append(" ] }"); + JSONObject resp = + connection.executeStatement(ElasticConnection.GET, "vector/_mget", buffer.toString()); + JSONArray docs = (JSONArray) resp.get("docs"); + char[] vectorDecodeBuffer = Base64VectorFactory.allocateBuffer(); + for (int i = 0; i < maxDocuments; ++i) { + if (!iter2.hasNext()) { + break; + } + vecRes = iter2.next(); + JSONObject oneResp = (JSONObject) docs.get(i); + String matchId = (String) oneResp.get("_id"); + long matchIdVal = Base64Lite.decodeLongBase64(matchId); + JSONObject source = (JSONObject) oneResp.get("_source"); + if (source == null) { + throw new ElasticException("vector document does not exist for id=" + matchId); + } + if (matchIdVal != vecRes.vectorid) { + throw new ElasticException("Mismatch in vectorid"); + } + StringReader reader = new StringReader((String) source.get("features")); + try { + vecRes.vec = vectorFactory.restoreVectorFromBase64(reader, vectorDecodeBuffer); + } + catch (IOException e) { + throw new ElasticException(e.getMessage()); + } + } + } + + /** + * Given a list of FunctionDescriptions, fill in the matching SignatureRecords + * @param listFunctions is the list of functions + * @param manager is the FunctionDescription container + * @throws ElasticException for communication problems with the server + */ + private void queryAssociatedSignatures(List listFunctions, + DescriptionManager manager) throws ElasticException { + TreeMap vecMap = new TreeMap<>(); + for (FunctionDescription fdesc : listFunctions) { // Collect (unique) vectorids across FunctionDescriptions + if (fdesc.getSignatureRecord() != null) { + continue; + } + Long key = Long.valueOf(fdesc.getVectorId()); + if (vecMap.containsKey(key)) { + continue; + } + VectorResult vecRes = new VectorResult(); + vecRes.vectorid = key.longValue(); + vecMap.put(key, vecRes); + } + Iterator iter1 = vecMap.values().iterator(); + Iterator iter2 = vecMap.values().iterator(); + while (iter1.hasNext()) { + fetchVectors(iter1, iter2, 50); // Fetch vector associated with each vectorid + } + iter1 = vecMap.values().iterator(); + iter2 = vecMap.values().iterator(); + while (iter1.hasNext()) { + fetchVectorCounts(iter1, iter2, MAX_VECTORCOUNT_WINDOW); // Fetch hitcount of each vector + } + TreeMap sigMap = new TreeMap<>(); + for (Entry entry : vecMap.entrySet()) { // Build SignatureRecord for each (id,vector,hitcount) + SignatureRecord sigRec = + manager.newSignature(entry.getValue().vec, entry.getValue().hitcount); + manager.setSignatureId(sigRec, entry.getKey()); + sigMap.put(entry.getKey(), sigRec); + } + for (FunctionDescription fdesc : listFunctions) { // Attach SignatureRecords to FunctionDescriptions + if (fdesc.getSignatureRecord() != null) { + continue; + } + SignatureRecord sigRec = sigMap.get(fdesc.getVectorId()); + manager.attachSignature(fdesc, sigRec); + } + } + + /** + * Query for function documents that match a given vector id and + * passes additional filters. The filter must already be encoded as a JSON fragment. + * @param vectorId is the vector id to match + * @param filter is the JSON fragment describing the filter + * @param maxDocuments is the maximum number of documents to return + * @return list of matching functions as JSON documents + * @throws ElasticException for communication problems with the server + */ + private JSONArray queryVectorIdMatch(long vectorId, String filter, int maxDocuments) + throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"size\": ").append(maxDocuments); + buffer.append(", \"_source\": { \"excludes\": [ \"childid\" ] }"); + buffer.append(", \"query\": { "); + buffer.append(" \"bool\": { "); + buffer.append(" \"must\": { "); + buffer.append(" \"term\": { \"id_signature\": \""); + Base64Lite.encodeLongBase64(buffer, vectorId); + buffer.append("\" } }"); + if (filter != null) { + buffer.append(filter); + } + buffer.append("} } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + JSONObject baseHits = (JSONObject) resp.get("hits"); + JSONObject totalRec = (JSONObject) baseHits.get("total"); + long total = (Long) totalRec.get("value"); + if (total == 0) { + return new JSONArray(); + } + JSONArray hitsArray = (JSONArray) baseHits.get("hits"); + return hitsArray; + } + + /** + * Query the database for all vectors that are "near" a given vector in terms + * of similarity and significance. The routine returns a list of VectorResult + * objects with the id, LSHVector, and hitcount filled in. + * @param listResult is the discovered list of VectorResults + * @param vector is the vector being queried + * @param similarityThreshold is the similarity threshold results must exceed + * @param significanceThreshold is the significance threshold results must exceed + * @param maxVectors is the maximum number of (distinct) vectors to return + * @return the total number of functions matching one of the returned vectors + * @throws ElasticException for communication problems with the server + */ + private long queryNearestVector(List listResult, LSHVector vector, + double similarityThreshold, double significanceThreshold, int maxVectors) + throws ElasticException { + if (connection == null) { + return 0; + } + StringBuilder vecEncode = new StringBuilder(); + vector.saveBase64(vecEncode, Base64Lite.encode); + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"size\": ").append(maxVectors).append(", "); + buffer.append(" \"query\": { "); + buffer.append(" \"function_score\": { "); + buffer.append(" \"query\": { "); + buffer.append(" \"match\": { "); + buffer.append(" \"features\": \""); + buffer.append(vecEncode); + buffer.append("\" } }, "); + buffer.append(" \"min_score\": 0.00001, "); // Make sure a 0.0 score is filtered + buffer.append(" \"boost_mode\": \"replace\", "); + buffer.append(" \"functions\": [ { "); + buffer.append(" \"script_score\": { "); + buffer.append(" \"script\": { "); + buffer.append(" \"lang\": \"bsim_scripts\", "); + buffer.append(" \"source\": \"lsh_compare\", "); + buffer.append(" \"params\": { "); + buffer.append(" \"indexname\": \"lsh_").append(repository).append("\", "); + buffer.append(" \"vector\": \""); + buffer.append(vecEncode); + buffer.append("\", \"simthresh\": ").append(similarityThreshold); + buffer.append(", \"sigthresh\": ").append(significanceThreshold); + buffer.append(" } } } } ] } } }"); + JSONObject resp = + connection.executeStatement(ElasticConnection.GET, "vector/_search", buffer.toString()); + JSONObject baseHits = (JSONObject) resp.get("hits"); + if (baseHits == null) { + throw new ElasticException("Could not find hits document"); + } + JSONObject totalRec = (JSONObject) baseHits.get("total"); + long numHits = (Long) totalRec.get("value"); + if (numHits == 0) { + return 0; + } + JSONArray hitsArray = (JSONArray) baseHits.get("hits"); + char[] decodeBuffer = Base64VectorFactory.allocateBuffer(); + VectorCompare vecCompare = new VectorCompare(); + try { + int returnedHits = hitsArray.size(); + for (int i = 0; i < returnedHits; ++i) { + JSONObject mainHit = (JSONObject) hitsArray.get(i); + VectorResult vecRes = new VectorResult(); + vecRes.vectorid = Base64Lite.decodeLongBase64((String) mainHit.get("_id")); + vecRes.hitcount = -1; // Cannot fill in at this time + vecRes.sim = (Double) mainHit.get("_score"); + JSONObject source = (JSONObject) mainHit.get("_source"); + StringReader reader = new StringReader((String) source.get("features")); + vecRes.vec = vectorFactory.restoreVectorFromBase64(reader, decodeBuffer); + vector.compareCounts(vecRes.vec, vecCompare); + vecCompare.dotproduct = vecRes.sim * vector.getLength() * vecRes.vec.getLength(); + vecRes.signif = vectorFactory.calculateSignificance(vecCompare); + listResult.add(vecRes); + } + } + catch (IOException ex) { + throw new ElasticException("Bad encoding in result document"); + } + long totalCount = 0; + Iterator iter1 = listResult.iterator(); + Iterator iter2 = listResult.iterator(); + while (iter1.hasNext()) { + totalCount += fetchVectorCounts(iter1, iter2, MAX_VECTORCOUNT_WINDOW); + } + return totalCount; + } + + /** + * Returns the total number of hits in the given list of VectorResults + * + * @param listResult is the list of VectorResults + * @return the total count + */ + private int getTotalCount(List listResult) { + int count = 0; + for (VectorResult res : listResult) { + count += res.hitcount; + } + return count; + } + + /** + * Query the database for functions that are similar to the given feature vector + * @param similarityResult receives the list of results and their similarity to the base vector + * @param manager is the DescriptionManager container for the results + * @param vector is the feature vector to match + * @param query contains various thresholds for the query + * @param filter specifies additional conditions functions (and exes) must meet after meeting sim/signif threshold + * @param vecToResultsMap is a cache of VectorResult lists from previous queries + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding new records to the container + */ + private void queryNearest(SimilarityResult similarityResult, DescriptionManager manager, + LSHVector vector, QueryNearest query, String filter, + HashMap> vecToResultsMap) + throws ElasticException, LSHException { + List resultset = new ArrayList<>(); + int vectormax = query.vectormax; + if (vectormax == 0) { + vectormax = MAX_VECTOR_OVERALL; // Really means a very big limit + } + + // Check to see if we've already queried for this vector before. If so, just grab + // the results from the map. + if (vecToResultsMap.containsKey(vector)) { + resultset = vecToResultsMap.get(vector); + similarityResult.setTotalCount(getTotalCount(resultset)); + } + else { + // Perform the query. + similarityResult.setTotalCount((int) queryNearestVector(resultset, vector, query.thresh, + query.signifthresh, vectormax)); + + // Put the new results in the map so we can use them if another + // similar vector comes along. + vecToResultsMap.put(vector, resultset); + } + + int count = 0; + + for (VectorResult dresult : resultset) { + if (count >= query.max) { + break; + } + final SignatureRecord srec = manager.newSignature(dresult.vec, dresult.hitcount); + JSONArray descres; + descres = queryVectorIdMatch(dresult.vectorid, filter, query.max - count); + if (descres == null) { + throw new ElasticException( + "Error querying vectorid: " + Long.toString(dresult.vectorid)); + } + if (descres.size() == 0) { + if (filter != null) { + continue; // Filter may have eliminated all results + } + // Otherwise this is a sign of corruption in the database + throw new ElasticException( + "No functions matching vectorid: " + Long.toString(dresult.vectorid)); + } + count += descres.size(); + convertDescriptionRows(similarityResult, descres, dresult, manager, srec); + } + } + + /** + * Perform a full QueryNearest request, with additional filters, placing SimilarityResults + * in the ResponseNearest object. An iterator to FunctionDescriptions determines what + * subset of functions are actually being queried. + * @param query is overarching QueryNearest object + * @param filter is the (optional) additional filters results must pass + * @param response is the ResponseNearest accumulating results + * @param manager is an internal placeholder container primarily for caching ExecutableRecords + * @param iter points to the subset of functions to query + * @return the total number of unique result sets produced by the query + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding new records to the response + */ + private int queryFunctions(QueryNearest query, String filter, ResponseNearest response, + DescriptionManager manager, Iterator iter) + throws ElasticException, LSHException { + + // Keep a map of the feature vectors and their query results; if we have vectors that + // are of equal value, we'll just query the first one, and use those results for the + // others. + HashMap> vecToResultMap = new HashMap<>(); + + while (iter.hasNext()) { + final FunctionDescription frec = iter.next(); + final SignatureRecord srec = frec.getSignatureRecord(); + + if (srec == null) { + continue; + } + final LSHVector thevec = srec.getLSHVector(); + final double len2 = vectorFactory.getSelfSignificance(thevec); + + // Self significance should be bigger than the significance threshold + // (or its impossible our result can exceed the threshold) + if (len2 < query.signifthresh) { + continue; + } + response.totalfunc += 1; + final SimilarityResult simres = new SimilarityResult(frec); + if (manager.getExecutableRecordSet().size() > 1000) { + manager.clear(); + } + else { + // Try to preserve ExecutableRecords so we don't have to connect every time + manager.clearFunctions(); + } + queryNearest(simres, manager, thevec, query, filter, vecToResultMap); + if (simres.size() == 0) { + continue; + } + response.totalmatch += 1; + if (simres.size() == 1) { + response.uniquematch += 1; + } + + response.result.add(simres); + + simres.transfer(response.manage, true); + } + + return vecToResultMap.size(); + } + + /** + * Query for function names within a previously queried executable + * @param listFunctions - list of functions to be filled in by the query (may be null) + * @param manager - container for record + * @param exeRecord - previously queried ExecutableRecord + * @param functionName - name to query for, if empty string, all functions in executable are returned + * @param fillInSignatures - true if SignatureRecords should be filled in for resulting FunctionDescriptions + * @param maxFunctions - maximum results to return + * @throws ElasticException for communication problems with the server + */ + private void queryByName(List listFunctions, DescriptionManager manager, + ExecutableRecord exeRecord, String functionName, boolean fillInSignatures, + int maxFunctions) throws ElasticException { + RowKeyElastic eKey = (RowKeyElastic) exeRecord.getRowId(); + String exeId = eKey.generateExeIdString(); + if (listFunctions == null) { + listFunctions = new ArrayList<>(); + } + if (functionName.length() == 0) { + queryAllFunc(listFunctions, exeRecord, exeId, manager, maxFunctions); + } + else { + JSONArray hitsarray = queryFuncNameMatch(exeId, functionName, maxFunctions); + JSONObject doc = null; + for (Object element : hitsarray) { + doc = (JSONObject) element; + FunctionDescription funcDesc = convertDescriptionRow(doc, exeRecord, manager, null); + listFunctions.add(funcDesc); + } + } + if (fillInSignatures) { + queryAssociatedSignatures(listFunctions, manager); + } + } + + /** + * Given an ExecutableRecord, function name, and address, query for + * the matching FunctionDescription. + * @param manager is the container for the FunctionDescription + * @param exeRecord is the given executable + * @param functionName is the function name + * @param address is the function address + * @param fillInSignatures is true if the SignatureRecord should be filled in + * @return the recovered FunctionDescription or null if not found + * @throws ElasticException for communication problems with the server + */ + private FunctionDescription queryByNameAddress(DescriptionManager manager, + ExecutableRecord exeRecord, String functionName, long address, boolean fillInSignatures) + throws ElasticException { + RowKeyElastic eKey = (RowKeyElastic) exeRecord.getRowId(); + String exeId = eKey.generateExeIdString(); + JSONObject doc = queryFuncNameAddress(exeId, functionName, address); + if (doc == null) { + return null; + } + FunctionDescription funcDesc = convertDescriptionRow(doc, exeRecord, manager, null); + if (fillInSignatures) { + List vecres = new ArrayList<>(); + vecres.add(funcDesc); + queryAssociatedSignatures(vecres, manager); + } + return funcDesc; + } + + /** + * Retrieve a sequence of ExecutableRecords by id. The ids are queried in bulk, + * up to a given maximum number of documents to fetch. Two iterators pointing to + * the same list of RowKeys must be provided. One is used while building the query + * document. The other is used to assign the matching RowKey to the new ExecutableRecords + * @param manager is the container to store new ExecutableRecord results + * @param iter1 is an iterator over ids + * @param iter2 is a copy of the iterator over ids + * @param maxDocuments is the maximum number of ids to fetch in this request + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding new records to the container + */ + private void queryExecutableRecordById(DescriptionManager manager, + Iterator iter1, Iterator iter2, int maxDocuments) + throws ElasticException, LSHException { + StringBuilder buffer = new StringBuilder(); + if (!iter1.hasNext()) { + return; // Nothing to do + } + String path = '/' + repository + "_executable/_msearch"; + int count = 0; + for (int i = 0; i < maxDocuments; ++i) { + String exeId = iter1.next().generateExeIdString(); + buffer.append("{}\n"); // Keep default index and type + buffer.append("{ \"query\": { \"bool\": { \"filter\": { \"term\": { \"_id\": \""); + buffer.append(exeId); + buffer.append("\" }}}}}\n"); + count += 1; + if (!iter1.hasNext()) { + break; + } + } + JSONObject bulkobj = connection.executeBulk(path, buffer.toString()); + JSONArray responses = (JSONArray) bulkobj.get("responses"); + for (int i = 0; i < count; ++i) { + JSONObject subquery = (JSONObject) responses.get(i); + JSONObject hits = (JSONObject) subquery.get("hits"); + if (hits == null) { + throw new ElasticException("Multi-search for exe records failed"); + } + JSONObject totalRec = (JSONObject) hits.get("total"); + long total = (Long) totalRec.get("value"); + if (total != 1) { + throw new ElasticException("Could not recover unique executable via id"); + } + } + for (int i = 0; i < count; ++i) { + JSONObject subquery = (JSONObject) responses.get(i); + JSONObject hits = (JSONObject) subquery.get("hits"); + JSONArray hitsArray = (JSONArray) hits.get("hits"); + hits = (JSONObject) hitsArray.get(0); + ExecutableRecord newExe = makeExecutableRecord(manager, hits); + RowKey rowKey = iter2.next(); + manager.cacheExecutableByRow(newExe, rowKey); + } + } + + /** + * Query for function documents based on their parent executable id. + * A "page" of results is selected by selecting a -start- document and a maximum number to return + * @param exeId is the executable id + * @param maxDocuments is the maximum number of functions to return + * @param start is the number functions to skip + * @return the JSON response object + * @throws ElasticException for communication problems with the server + */ + private JSONObject queryFunctionsOfExeId(String exeId, long maxDocuments, long start) + throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"size\": ").append(maxDocuments); + buffer.append(", \"_source\": { \"excludes\": [ \"childid\" ] }"); + buffer.append(", \"query\": { \"parent_id\": { \"type\": \"function\", \"id\": \"") + .append(exeId) + .append("\" }}"); + if (start != 0) { + buffer.append(", \"search_after\": [").append(start).append(']'); + } + buffer.append(", \"sort\": [ { \"_doc\": \"asc\" } ] }"); + return connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + } + + /** + * Query for all functions (up to a maximum) of the given executable. + * Populate a list with the resulting FunctionDescription objects + * Does NOT populate SignatureRecord or CallGraph parts of the FunctionDescription + * @param listFunctions has all queried functions added to it + * @param exeRecord is the executable containing the functions + * @param exeId is a precomputed document id of the executable + * @param manager is the container for the new records + * @param maxDocuments is the maximum number of records to read + * @return the number of function records read + * @throws ElasticException for communication problems with the server + */ + private int queryAllFunc(List listFunctions, ExecutableRecord exeRecord, + String exeId, DescriptionManager manager, int maxDocuments) throws ElasticException { + long total; + long start = 0; + do { + int limit = MAX_FUNCTION_WINDOW; + if (maxDocuments != 0 && maxDocuments - start < limit) { + limit = (int) (maxDocuments - start); + } + JSONObject resp = queryFunctionsOfExeId(exeId, limit, start); + JSONObject hits = (JSONObject) resp.get("hits"); + JSONObject totalRec = (JSONObject) hits.get("total"); + total = (Long) totalRec.get("value"); + if (maxDocuments != 0 && maxDocuments < total) { + total = maxDocuments; + } + JSONArray hitsarray = (JSONArray) hits.get("hits"); + JSONObject doc = null; + for (Object element : hitsarray) { + doc = (JSONObject) element; + FunctionDescription funcDesc = convertDescriptionRow(doc, exeRecord, manager, null); + listFunctions.add(funcDesc); + } + if (hitsarray.size() == 0) { + break; // Shouldn't need this, but just in case + } + JSONArray sort = (JSONArray) doc.get("sort"); + start = (Long) sort.get(0); // Sort value for last entry, for passing as search_after parameter + } + while (total > start); + return (int) total; + } + + /** + * Issue a bulk update request to the database, given a list of update records + * for functions from a single executable. + * The routine needs an iterator to the FunctionDescription.Updates and only processes up to + * a given maximum number in the one bulk request. + * @param iter is the iterator to the update records + * @param exeId is the document id of the executable containing the functions + * @param maxFunctions is the maximum number of updates to put in the one request + * @throws ElasticException for communication problems with the server + */ + private void updateFunctions(Iterator iter, String exeId, + int maxFunctions) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + do { + FunctionDescription.Update rec = iter.next(); + RowKeyElastic eKey = (RowKeyElastic) rec.update.getExecutableRecord().getRowId(); + buffer.append("{ \"update\": { \"_index\": \"") + .append(repository) + .append("_executable\", "); + buffer.append("\"_id\": \""); + eKey.generateFunctionId(buffer, rec.update); + buffer.append("\", \"routing\": \"").append(exeId).append("\"} }\n"); + buffer.append("{ \"doc\": { "); + boolean needscomma = false; + if (rec.function_name) { + needscomma = true; + buffer.append("\"name_func\": \"") + .append(JSONObject.escape(rec.update.getFunctionName())) + .append('\"'); + } + if (rec.flags) { + if (needscomma) { + buffer.append(','); + } + buffer.append(" \"flags\": ").append(rec.update.getFlags()); + } + buffer.append("} }\n"); + } + while (iter.hasNext()); + connection.executeBulk("/_bulk", buffer.toString()); + } + + /** + * Issue an update request for single executable, given its document id and an update record + * @param updateRecord is the executable specific update record + * @param exeId is the document id of the executable + * @throws ElasticException for communication problems with the server + */ + private void updateExecutable(ExecutableRecord.Update updateRecord, String exeId) + throws ElasticException { + StringBuilder buffer = new StringBuilder(); + boolean needscomma = false; + buffer.append("{ \"doc\": {"); + if (updateRecord.name_exec) { + needscomma = true; + buffer.append("\"name_exec\": \"") + .append(JSONObject.escape(updateRecord.update.getNameExec())) + .append('\"'); + } + if (updateRecord.architecture) { + if (needscomma) { + buffer.append(','); + } + else { + needscomma = true; + } + buffer.append("\"architecture\": \"") + .append(updateRecord.update.getArchitecture()) + .append('\"'); + } + if (updateRecord.name_compiler) { + if (needscomma) { + buffer.append(','); + } + else { + needscomma = true; + } + buffer.append("\"name_compiler\": \"") + .append(updateRecord.update.getArchitecture()) + .append('\"'); + } + if (updateRecord.date) { + if (needscomma) { + buffer.append(','); + } + else { + needscomma = true; + } + buffer.append("\"ingest_date\": ").append(updateRecord.update.getDate().getTime()); + } + if (updateRecord.repository) { + if (needscomma) { + buffer.append(','); + } + else { + needscomma = true; + } + buffer.append("\"repository\": \"") + .append(updateRecord.update.getRepository()) + .append('\"'); + } + if (updateRecord.path) { + if (needscomma) { + buffer.append(','); + } + else { + needscomma = true; + } + buffer.append("\"path\": \"").append(updateRecord.update.getPath()).append('\"'); + } + if (updateRecord.categories) { + if (needscomma) { + buffer.append(','); + } + else { + needscomma = true; + } + appendCategoryTag(updateRecord.update.getAllCategories(), buffer); + } + buffer.append("} }"); + StringBuilder pathBuffer = new StringBuilder(); + pathBuffer.append("executable/_update/"); + pathBuffer.append(exeId); + connection.executeStatementNoResponse(ElasticConnection.POST, pathBuffer.toString(), + buffer.toString()); + } + + /** + * Update metadata for the executable -erec- and all its functions (in manager) + * @param manager is the collection of functions to update + * @param exeRecord is the root executable + * @param badFunctions collects references to functions with update info that could not be identified + * @return -1 if the executable could not be found, otherwise return 2*# of + * update functions + 1 if the executable metadata is also updated + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems grouping records + */ + private int updateExecutable(DescriptionManager manager, ExecutableRecord exeRecord, + List badFunctions) throws ElasticException, LSHException { + JSONObject row = queryMd5ExeMatch(exeRecord.getMd5()); + if (row == null) { + return -1; // Indicate that we couldn't find the executable + } + ExecutableRecord erec_db = makeExecutableRecordTemp(row); + DescriptionManager dbmanage = new DescriptionManager(); + erec_db = dbmanage.transferExecutable(erec_db); + ExecutableRecord.Update exe_update = new ExecutableRecord.Update(); + boolean has_exe_update = exeRecord.diffForUpdate(exe_update, erec_db); + + // Load all the functions in the database under this executable + RowKeyElastic eKey = (RowKeyElastic) erec_db.getRowId(); + String exeId = eKey.generateExeIdString(); + List funclist = new ArrayList<>(); + queryAllFunc(funclist, erec_db, exeId, dbmanage, 0); + + // Create a map from address to executables + Map addrmap = + FunctionDescription.createAddressToFunctionMap(funclist.iterator()); + + // Match new functions to old functions via the address + List updatelist; + updatelist = FunctionDescription.generateUpdates(manager.listFunctions(exeRecord), addrmap, + badFunctions); + + if (!has_exe_update && updatelist.isEmpty()) { + return 0; // All updates are in place already + } + + // Do the actual database updates + if (has_exe_update) { + updateExecutable(exe_update, exeId); + } + Iterator iter = updatelist.iterator(); + while (iter.hasNext()) { + updateFunctions(iter, exeId, MAX_FUNCTIONUPDATE_WINDOW); + } + int val = has_exe_update ? 1 : 0; + val += 2 * updatelist.size(); + return val; + } + + /** + * Create a list of CategoryRecord objects from an "exe" JSON document, + * by extracting all the "execategory" properties from the document + * @param source is the exe document + * @return the list of CategoryRecords + */ + private static List makeCategoryList(JSONObject source) { + JSONArray catArray = (JSONArray) source.get("execategory"); + if (catArray == null || catArray.size() == 0) { + return null; + } + List res = new ArrayList<>(); + for (Object element : catArray) { + String concat = (String) element; + int pos = concat.indexOf('\t'); + if (pos > 0) { + String type = concat.substring(0, pos); + String value = concat.substring(pos + 1); + res.add(new CategoryRecord(type, value)); + } + } + return res; + } + + /** + * Create an ExecutableRecord from a JSON "hit" document returned when querying the executable/exe index + * The record will not be attached to any container (DescriptionManager), although it + * can be transferred into a container later. + * @param hit is the "hit" document, which should have an "_id" and "_source" property. + * @return the new ExecutableRecord parsed from the document + */ + private static ExecutableRecord makeExecutableRecordTemp(JSONObject hit) { + RowKeyElastic eKey = RowKeyElastic.parseExeIdString((String) hit.get("_id")); + JSONObject source = (JSONObject) hit.get("_source"); + String md5 = (String) source.get("md5"); + String exename = (String) source.get("name_exec"); + String arch = (String) source.get("architecture"); + ExecutableRecord exeres; + if (ExecutableRecord.isLibraryHash(md5)) { + exeres = new ExecutableRecord(exename, arch, eKey); + } + else { + String cname = (String) source.get("name_compiler"); + String repo = (String) source.get("repository"); + String path = null; + if (repo != null) { + path = (String) source.get("path"); + } + List catrecs = makeCategoryList(source); + long milli = (Long) source.get("ingest_date"); + exeres = new ExecutableRecord(md5, exename, cname, arch, new Date(milli), catrecs, eKey, + repo, path); + } + return exeres; + } + + /** + * Create an ExecutableRecord from a JSON "hit" document returned when querying the executable/exe index + * @param manager is the container that will own the new record + * @param hit is the "hit" document, which should have an "_id" and "_source" property. + * @return the new ExecutableRecord parsed from the document + * @throws LSHException if the container already contains executable with different metadata + */ + private static ExecutableRecord makeExecutableRecord(DescriptionManager manager, JSONObject hit) + throws LSHException { + RowKeyElastic eKey = RowKeyElastic.parseExeIdString((String) hit.get("_id")); + JSONObject source = (JSONObject) hit.get("_source"); + String md5 = (String) source.get("md5"); + String exename = (String) source.get("name_exec"); + String arch = (String) source.get("architecture"); + + ExecutableRecord exerec; + if (ExecutableRecord.isLibraryHash(md5)) { + exerec = manager.newExecutableLibrary(exename, arch, eKey); + } + else { + String cname = (String) source.get("name_compiler"); + String repo = (String) source.get("repository"); + String path = null; + if (repo != null) { + path = (String) source.get("path"); + } + List catrecs = makeCategoryList(source); + long milli = (Long) source.get("ingest_date"); + Date date = new Date(milli); + exerec = manager.newExecutableRecord(md5, exename, cname, arch, date, repo, path, eKey); + if (catrecs != null) { + manager.setExeCategories(exerec, catrecs); + } + } + return exerec; + } + + /** + * Build a FunctionDescription object in -manager- container from a -hit- document + * returned from a query into the executable/function index. + * @param hit the hit document + * @param exeRecord is the executable containing this function + * @param manager is the DescriptionManager container + * @param sigRecord is the (optional) SignatureRecord to attach to the new function + * @return the new FunctionDescription + */ + private static FunctionDescription convertDescriptionRow(JSONObject hit, + ExecutableRecord exeRecord, DescriptionManager manager, SignatureRecord sigRecord) { + RowKey rowid = RowKeyElastic.parseFunctionId((String) hit.get("_id")); + JSONObject source = (JSONObject) hit.get("_source"); + String func_name = (String) source.get("name_func"); + long addr = (Long) source.get("addr"); + int flags = ((Long) source.get("flags")).intValue(); + long id_sig = Base64Lite.decodeLongBase64((String) source.get("id_signature")); + FunctionDescription fres = manager.newFunctionDescription(func_name, addr, exeRecord); + manager.setFunctionDescriptionId(fres, rowid); + manager.setFunctionDescriptionFlags(fres, flags); + manager.setSignatureId(fres, id_sig); + if (sigRecord != null) { + manager.setSignatureId(sigRecord, id_sig); + manager.attachSignature(fres, sigRecord); + } + return fres; + } + + /** + * Convert function documents, presented as an array of JSON objects, that all + * share a single feature vector returned by a nearest neighbor query, into + * FunctionDescriptions and a full SimilarityResult. + * Each function document will be parsed into a FunctionDescription, + * and a SimilarityNote will be created describing its similarity to + * the query vector based on the raw VectorResult data. + * @param similarityResult is the container for the SimilarityNotes + * @param descRows is the array of JSON function documents + * @param vectorResult is the raw vector query result + * @param manager is the container for new FunctionDescriptions + * @param sigRecord is the shared feature vector + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding new records to the container + */ + protected void convertDescriptionRows(SimilarityResult similarityResult, JSONArray descRows, + VectorResult vectorResult, DescriptionManager manager, SignatureRecord sigRecord) + throws ElasticException, LSHException { + if (descRows.size() == 0) { + return; + } + List parentIds = new ArrayList<>(descRows.size()); + Set parents = new TreeSet<>(); + RowKeyElastic eKey; + String exeid; + for (Object descRow : descRows) { + JSONObject hit = (JSONObject) descRow; + JSONObject source = (JSONObject) hit.get("_source"); + JSONObject joinfield = (JSONObject) source.get("join_field"); + exeid = (String) joinfield.get("parent"); + eKey = RowKeyElastic.parseExeIdString(exeid); + parentIds.add(eKey); + if (manager.findExecutableByRow(eKey) == null) { + parents.add(eKey); + } + } + Iterator iter1 = parents.iterator(); + Iterator iter2 = parents.iterator(); + while (iter1.hasNext()) { + queryExecutableRecordById(manager, iter1, iter2, 100); + } + + JSONObject currow = (JSONObject) descRows.get(0); + eKey = parentIds.get(0); + ExecutableRecord curexe = manager.findExecutableByRow(eKey); + FunctionDescription fres = convertDescriptionRow(currow, curexe, manager, sigRecord); + if (similarityResult != null) { + similarityResult.addNote(fres, vectorResult.sim, vectorResult.signif); + } + for (int i = 1; i < descRows.size(); ++i) { + currow = (JSONObject) descRows.get(i); + eKey = parentIds.get(i); + curexe = manager.findExecutableByRow(eKey); + fres = convertDescriptionRow(currow, curexe, manager, sigRecord); + if (similarityResult != null) { + similarityResult.addNote(fres, vectorResult.sim, vectorResult.signif); + } + } + } + + /** + * Insert a library executable and all the functions it contains into the database. + * The executable must be a library. The routine will complete successfully even if + * the executable or some of its functions have been inserted before. + * @param manager is container of the executable + * @param exeRecord is the ExecutableRecord describing the library + * @throws ElasticException for communication problems with the server + */ + private void insertLibrary(DescriptionManager manager, ExecutableRecord exeRecord) + throws ElasticException { + RowKeyElastic eKey = updateKey(manager, exeRecord); + String exeId = eKey.generateExeIdString(); + insertExecutableRecord(exeRecord, exeId); // May or may not already be inserted, doesn't matter + + Iterator iter = manager.listFunctions(exeRecord); + while (iter.hasNext()) { + insertFunctionRange(manager, exeRecord, eKey, exeId, iter, MAX_FUNCTION_BULK); // Insert functions some of which may already be inserted + } + } + + /** + * Insert an executable and all of the functions it contains into the database. + * The executable must not be a library. If the executable has been inserted before, + * a non-fatal exception is thrown if the previous executable and this new one have + * exactly matching meta-data. A fatal exception is thrown if meta-data has changed. + * All functions (FunctionDescriptions) associated with the executable are inserted, + * including their feature vectors. + * @param manager is the container of the executable + * @param exeRecord is the executable to insert + * @return false only if the insert was unsuccessful but a previous executable could not be recovered + * @throws ElasticException for communication problems with the server + * @throws LSHException if update differences cannot reconciled + * @throws DatabaseNonFatalException for non-fatal updates that can't be executed + */ + private boolean insertExe(DescriptionManager manager, ExecutableRecord exeRecord) + throws ElasticException, LSHException, DatabaseNonFatalException { + RowKeyElastic eKey = updateKey(manager, exeRecord); + String exeId = eKey.generateExeIdString(); + + if (!insertExecutableRecord(exeRecord, exeId)) { // Try to insert the executable + JSONObject exeObj = queryMd5ExeMatch(exeRecord.getMd5()); + if (exeObj != null) { // Try to retrieve the previous version + ExecutableRecord oldrec = makeExecutableRecordTemp(exeObj); + int cmp = oldrec.compareMetadata(exeRecord); + if (cmp != 0) { + String fatalerror = + FunctionDatabase.constructFatalError(cmp, exeRecord, oldrec); + if (fatalerror != null) { + throw new LSHException(fatalerror); + } + throw new DatabaseNonFatalException( + FunctionDatabase.constructNonfatalError(cmp, exeRecord, oldrec)); + } + throw new DatabaseNonFatalException( + exeRecord.getNameExec() + " is already ingested"); + } + return false; // Indicate this executable already inserted + } + int newIds = 0; + long baseId = 0; + Iterator iter = manager.listFunctions(exeRecord); + while (iter.hasNext()) { // Count the functions to insert for this executable + iter.next(); + newIds += 1; + } + baseId = allocateFunctionIndexSpace(newIds); // Allocated the ids we will need + iter = manager.listFunctions(exeRecord); + while (iter.hasNext()) { + manager.setFunctionDescriptionId(iter.next(), new RowKeyElastic(baseId)); // Set the (allocated) ids + baseId += 1; + } + // Collect/dedup vectors and update SignatureRecords with vector ids, before writing FunctionDescriptions + Set vectorContainer = + IdHistogram.collectVectors(manager, manager.listFunctions(exeRecord)); + + iter = manager.listFunctions(exeRecord); + // Write the FunctionDescriptions now that vector ids are filled in + while (iter.hasNext()) { + insertFunctionRange(manager, exeRecord, eKey, exeId, iter, MAX_FUNCTION_BULK); + } + + // Create/update the vector documents + Iterator viter = vectorContainer.iterator(); + while (viter.hasNext()) { + insertVectorRange(viter, MAX_VECTOR_BULK); + } + return true; + } + + /** + * Create configuration index, containing the keyvalue pairs + * and the sequence counter for ExecutableRecord/FunctionDescription document ids + * @throws ElasticException for communication problems with the server + */ + private void createConfigurationIndex() throws ElasticException { + StringBuilder builder = new StringBuilder(); + builder.append("{ \"settings\": { "); + builder.append(" \"number_of_shards\": 1, "); + builder.append(" \"auto_expand_replicas\": \"0-all\" }, "); + builder.append(" \"mappings\": { "); + builder.append(" \"dynamic\": \"strict\", "); + builder.append(" \"properties\": { "); + builder.append(" \"type\": { "); // Can be "sequence" or "keyvalue" + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"iid\": { "); + builder.append(" \"type\": \"long\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"value\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false } } } }"); + connection.executeStatementNoResponse(ElasticConnection.PUT, "configuration", + builder.toString()); + + // Make sure initial counter document exists and initializes counter to 1 + connection.executeStatementNoResponse(ElasticConnection.PUT, "configuration/_doc/1", + "{ \"type\": \"sequence\", \"iid\": 1 }"); + } + + /** + * Allocate a specific number of ids for function documents. The document id for a function + * is an integer unique across the entire database. Allocation of this id is implemented as an integer + * counter stored in a single document. Updating this document for an allocation must take into account + * distributed nodes simultaneously requesting ids. + * @param amount is the number of ids to allocate + * @return the first integer in the newly allocated set of ids + * @throws ElasticException for communication problems with the server + */ + private long allocateFunctionIndexSpace(int amount) throws ElasticException { + String body = + "{ \"script\": { \"inline\": \"ctx._source.iid += params.bulk_size\", \"params\": { \"bulk_size\": " + + Integer.toString(amount) + "}}}"; + JSONObject resp = connection.executeStatement(ElasticConnection.POST, + "configuration/_update/1?_source=iid&retry_on_conflict=5", body); + JSONObject get = (JSONObject) resp.get("get"); + JSONObject fields = (JSONObject) get.get("_source"); + long res = (Long) fields.get("iid"); + return res - amount; + } + + /** + * Insert a set of vector documents. Update/create the corresponding meta documents that + * count the number of times unique vectors are multiply inserted. Take into account simultaneous + * updates from distributed nodes. The vectors are presented as an iterator to IdHistogram, with + * the vectors already locally deduped and counted. This routine submits one bulk insertion/update + * request, including unique vectors only up to a specified maximum. The iterator is advanced for + * each unique vector included with the one request. + * @param iter is the iterator to unique vectors and counts (IdHistogram) + * @param maxVectors is the maximum number of unique vectors to include in this one request + * @throws ElasticException for communication problems with the server + */ + private void insertVectorRange(Iterator iter, int maxVectors) + throws ElasticException { + StringBuilder buffer = new StringBuilder(); + do { + IdHistogram entry = iter.next(); + // Create the document _vector/vector if it doesn't already exist + buffer.append("{ \"create\": { \"_index\": \"") + .append(repository) + .append("_vector\", "); + buffer.append("\"_id\": \""); + Base64Lite.encodeLongBase64(buffer, entry.id); + buffer.append("\" } }\n"); + buffer.append("{ \"features\": \""); + entry.vec.saveBase64(buffer, Base64Lite.encode); + buffer.append("\" }\n"); + // Upsert the counter document _vector/meta + buffer.append("{ \"update\": { \"_index\": \"").append(repository).append("_meta\", "); + buffer.append("\"_id\": \""); + Base64Lite.encodeLongBase64(buffer, entry.id); + buffer.append("\", \"retry_on_conflict\": 5 } }\n"); + buffer.append("{ \"script\": { \"inline\": \"ctx._source.count += params.count\", "); + buffer.append("\"params\": { \"count\": ").append(entry.count).append("} },"); + buffer.append("\"upsert\": { \"count\": ").append(entry.count).append("} }\n"); + maxVectors -= 1; + if (maxVectors <= 0) { + break; + } + } + while (iter.hasNext()); + JSONObject resp = connection.executeBulk("/_bulk", buffer.toString()); + if ((Boolean) resp.get("errors")) { + JSONArray items = (JSONArray) resp.get("items"); + for (Object item2 : items) { + JSONObject item = (JSONObject) item2; + JSONObject create = (JSONObject) item.get("create"); + JSONObject error = null; + if (create != null) { + error = (JSONObject) create.get("error"); + if (error != null) { + String type = (String) error.get("type"); + if (type.startsWith("version_conflict")) { + continue; // Normal error, meaning document already exists + } + } + } + else { + JSONObject update = (JSONObject) item.get("update"); + error = (JSONObject) update.get("error"); + } + if (error != null) { + throw new ElasticException((String) error.get("reason")); + } + } + } + } + + /** + * Decrement the "meta" document counter by the histogram count for a set of vectors. If any counter reaches zero, + * delete the meta document and add the vector record to the list of vectors scheduled for full deletion. + * @param deleteList accumulates records of counters that have reached zero + * @param iter1 is an iterator over records indicating the vector id and the count to decrement by + * @param iter2 is a copy of the first iterator + * @param maxVectors is the maximum number of counter updates to issue in this window + * @throws ElasticException for communication problems with the server + */ + private void decrementVectorCounters(List deleteList, Iterator iter1, + Iterator iter2, int maxVectors) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + for (int i = 0; i < maxVectors; ++i) { + if (!iter1.hasNext()) { + break; + } + IdHistogram entry = iter1.next(); + buffer.append("{ \"update\": { \"_index\": \"").append(repository).append("_meta\", "); + buffer.append("\"_id\": \""); + Base64Lite.encodeLongBase64(buffer, entry.id); + buffer.append("\", \"retry_on_conflict\": 5 } }\n"); + buffer.append( + "{ \"script\": { \"inline\": \"if ((ctx._source.count -= params.count) <=0) { ctx.op = \\\"delete\\\" }\", "); + buffer.append("\"params\": { \"count\": ").append(entry.count).append("} } }\n"); + maxVectors -= 1; + } + JSONObject resp = connection.executeBulk("/_bulk", buffer.toString()); + JSONArray items = (JSONArray) resp.get("items"); + for (int i = 0; i < maxVectors; ++i) { + if (!iter2.hasNext()) { + break; + } + IdHistogram entry = iter2.next(); + JSONObject item = (JSONObject) items.get(i); + JSONObject update = (JSONObject) item.get("update"); + long id = Base64Lite.decodeLongBase64((String) update.get("_id")); + if (id != entry.id) { + throw new ElasticException("Mismatch in decrementVectorCounters"); + } + if ("deleted".equals(update.get("result"))) { + deleteList.add(entry); // Mark this vector for full deletion + } + } + } + + /** + * Delete vector documents in bulk. This assumes multiplicity counts in the "meta" documents + * have already been checked, and these vectors are scheduled for full document deletion. + * Vectors are presented as an iterator to IdHistograms. One bulk deletion request is + * submitted containing vectors up to a given maximum number. The iterator is advanced by + * the number submitted + * @param iter is an iterator over records containing the id's to delete + * @param maxVectors is the maximum number to delete for this window + * @throws ElasticException for communication problems with the server + */ + private void deleteRawVectors(Iterator iter, int maxVectors) + throws ElasticException { + StringBuilder buffer = new StringBuilder(); + do { + IdHistogram entry = iter.next(); + buffer.append("{ \"delete\": { \"_index\": \"") + .append(repository) + .append("_vector\", "); + buffer.append("\"_id\": \""); + Base64Lite.encodeLongBase64(buffer, entry.id); + buffer.append("\" } }\n"); + maxVectors -= 1; + } + while (iter.hasNext() && maxVectors > 0); + JSONObject resp = connection.executeBulk("/_bulk", buffer.toString()); + if ((Boolean) resp.get("errors")) { + throw new ElasticException("Error during vector deletion"); + } + } + + /** + * Delete function documents and exe document associated with an executable id + * @param exeId is the executable's document id + * @return the number of function documents deleted + * @throws ElasticException for communication problems with the server + */ + private int deleteExeDocuments(String exeId) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"query\": {"); + buffer.append(" \"parent_id\": {"); + buffer.append(" \"type\": \"function\","); + buffer.append(" \"id\": \"").append(exeId).append("\" } } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.POST, + "executable/_delete_by_query", buffer.toString()); + long numDocs = (Long) resp.get("deleted"); + connection.executeURIOnly(ElasticConnection.DELETE, "executable/_doc/" + exeId); + return (int) numDocs; + } + + /** + * Append configuration WeightFactory weights to a JSON document in progress + * @param builder accumulates the encoded weights + * @param config is the Configuration + */ + private static void appendWeightSettings(StringBuilder builder, Configuration config) { + double[] weightArray = config.weightfactory.toArray(); + builder.append('\"').append(weightArray[0]); + for (int i = 1; i < weightArray.length; ++i) { + builder.append(' ').append(weightArray[i]); + } + builder.append("\" "); + } + + /** + * Append configuration IDFLookup hashes to a JSON document in progress + * @param builder accumulates the encoded hashes + * @param config is the Configuration + */ + private static void appendLookupSettings(StringBuilder builder, Configuration config) { + int[] intArray = config.idflookup.toArray(); + builder.append('\"').append(intArray[0]); + for (int i = 1; i < intArray.length; ++i) { + builder.append(' ').append(intArray[i]); + } + builder.append("\" "); + } + + /** + * Adjust the number of replicas and the refresh rate for the database. + * Different settings may make sense depending on whether the database is + * doing a large ingest or is currently only responding to queries + * @param index is the specific database index to adjust + * @param numReplicas is the number of replicas (data replication) requested + * @param refreshRateInSecs is the refresh rate requested + * @throws ElasticException for communication problems with the server + */ + private void adjustReplicaRefresh(String index, int numReplicas, int refreshRateInSecs) + throws ElasticException { + StringBuilder builder = new StringBuilder(); + builder.append("{ \"index\": { "); + builder.append(" \"number_of_replicas\": ").append(numReplicas).append(", "); + builder.append(" \"refresh_interval\": \""); + if (refreshRateInSecs < 1) { + builder.append("-1"); // Indicates that no refreshes are scheduled + } + else { + builder.append(refreshRateInSecs).append('s'); + } + builder.append("\" } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.PUT, index + "/_settings", + builder.toString()); + Boolean ack = (Boolean) resp.get("acknowledged"); + if (ack == null) { + throw new ElasticException( + "Unknown response trying to adjust number_of_replicas and refresh_interval"); + } + if (!ack) { + throw new ElasticException("Cluster did not accept settings for index: " + index); + } + } + + /** + * This routine establishes the schema for the "vector" and "meta" document types + * for a new database. It also sets up weights and hashes for the vector tokenizer (lsh_tokenizer). + * @param config contains database configuration info + * @throws ElasticException for communication problems with the server + */ + private void createVectorIndex(Configuration config) throws ElasticException { + StringBuilder builder = new StringBuilder(); + builder.append("{ \"settings\": { "); + builder.append(" \"index\": { "); + builder.append(" \"analysis\": { "); + builder.append(" \"tokenizer\": { "); + builder.append(" \"lsh_").append(repository).append("\": { "); + builder.append(" \"type\": \"lsh_tokenizer\", "); + builder.append(" \"").append(ElasticUtilities.LSH_WEIGHTS).append("\": "); + appendWeightSettings(builder, config); + builder.append(", \"").append(ElasticUtilities.IDF_CONFIG).append("\": "); + appendLookupSettings(builder, config); + builder.append(", \"") + .append(ElasticUtilities.K_SETTING) + .append("\": ") + .append(config.k); + builder.append(", \"") + .append(ElasticUtilities.L_SETTING) + .append("\": ") + .append(config.L); + builder.append(" } }, "); + builder.append(" \"analyzer\": { "); + builder.append(" \"lsh_analyzer\": { "); + builder.append(" \"type\": \"custom\", "); + builder.append(" \"tokenizer\": \"lsh_") + .append(repository) + .append("\" } } } } }, "); + builder.append(" \"mappings\": { "); + builder.append(" \"properties\": { "); + builder.append(" \"features\": { "); + builder.append(" \"type\": \"text\", "); + builder.append(" \"norms\": false, "); + builder.append(" \"index_options\": \"freqs\", "); + builder.append(" \"analyzer\": \"lsh_analyzer\" } } } }"); + connection.executeStatementNoResponse(ElasticConnection.PUT, "vector", builder.toString()); + + builder = new StringBuilder(); + builder.append("{ \"mappings\": { "); + builder.append(" \"properties\": { "); + builder.append(" \"count\": { "); + builder.append(" \"type\": \"integer\", "); + builder.append(" \"index\": false } } } }"); + connection.executeStatementNoResponse(ElasticConnection.PUT, "meta", builder.toString()); + } + + /** + * This routine establishes the schema for "exe" and "function" document types in a new database. + * @throws ElasticException for communication problems with the server + */ + private void createExecutableIndex() throws ElasticException { + StringBuilder builder = new StringBuilder(); + builder.append("{ \"mappings\": { "); + builder.append(" \"properties\": { "); + builder.append(" \"md5\": { "); // "exe" properties + builder.append(" \"type\": \"keyword\" }, "); + builder.append(" \"name_exec\": { "); + builder.append(" \"type\": \"keyword\" }, "); + builder.append(" \"architecture\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"name_compiler\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"ingest_date\": { "); + builder.append(" \"type\": \"date\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"repository\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"path\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"execategory\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"name_func\": { "); // "function" properties + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"doc_values\": false }, "); + builder.append(" \"id_signature\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"doc_values\": false }, "); + builder.append(" \"flags\": { "); + builder.append(" \"type\": \"integer\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"addr\": { "); + builder.append(" \"type\": \"long\", "); + builder.append(" \"doc_values\": false }, "); + builder.append(" \"childid\": { "); + builder.append(" \"type\": \"keyword\", "); + builder.append(" \"index\": false }, "); + builder.append(" \"join_field\": { "); // parent/child relation between exe and function + builder.append(" \"type\": \"join\", "); + builder.append(" \"relations\": { "); + builder.append(" \"exe\": \"function\" }, "); + builder.append(" \"eager_global_ordinals\": true } } } }"); + + connection.executeStatementNoResponse(ElasticConnection.PUT, "executable", + builder.toString()); + } + + /** + * Construct the database connection given a URL. The URL protocol must be http, and the URL + * path must contain exactly one element naming the particular repository on the server. + * @param baseURL is the http URL + * @throws MalformedURLException if the URL is malformed + */ + public ElasticDatabase(URL baseURL) throws MalformedURLException { + String fullURL = baseURL.toString(); + if (fullURL.startsWith("elastic:")) { + // https is the true protocol + fullURL = "https:" + fullURL.substring(8); + } + + String path = baseURL.getPath(); + if (!fullURL.endsWith(path)) { + throw new MalformedURLException("URL path must indicate the repository only"); + } + repository = path.substring(1); + this.serverInfo = + new BSimServerInfo(DBType.elastic, baseURL.getHost(), baseURL.getPort(), repository); + this.baseURL = fullURL.substring(0, fullURL.length() - path.length()); + + lastError = null; + info = null; + status = Status.Unconnected; + initialized = false; + } + + /** + * @return true if a connection has been successfully initialized + */ + public boolean isInitialized() { + return initialized; + } + + /** + * Read database configuration ("keyvalue" documents) into a key/value pair map. + * @return the populated map + * @throws ElasticException for communication problems with the server + * @throws NoDatabaseException if the database (as opposed to the server) does not exist + */ + private Map readKeyValues() throws ElasticException, NoDatabaseException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"size\": 500, \"query\": { \"match_all\": {} } }"); + JSONObject resp; + try { + resp = connection.executeStatement(ElasticConnection.GET, "configuration/_search", + buffer.toString()); + } + catch (ElasticException ex) { + if (ex.getMessage().contains("index_not_found_exception")) { + throw new NoDatabaseException("Database instance does not exist"); + } + throw ex; + } + JSONObject baseHits = (JSONObject) resp.get("hits"); + long total = 0; + if (baseHits != null) { + JSONObject totalRec = (JSONObject) baseHits.get("total"); + total = (Long) totalRec.get("value"); + } + if (total <= 1) { + throw new ElasticException("Unrecoverable error: Could not find configuration"); + } + HashMap res = new HashMap<>(); + JSONArray hits = (JSONArray) baseHits.get("hits"); + for (Object hit2 : hits) { + JSONObject hit = (JSONObject) hit2; + String key = (String) hit.get("_id"); + JSONObject source = (JSONObject) hit.get("_source"); + Object value = source.get("value"); + if (value == null) { + continue; // This might be the "sequence" document + } + res.put(key, (String) value); + } + return res; + } + + /** + * Given a critical key in the database configuration, return its corresponding value + * @param key is the configuration key + * @param keyValue is the key/value map + * @return the corresponding value + * @throws ElasticException if the key is missing + */ + private String getCriticalValue(String key, Map keyValue) + throws ElasticException { + String value = keyValue.get(key); + if (value == null) { + throw new ElasticException("Missing critical configuration value: " + key); + } + return value; + } + + /** + * Write DatabaseInformation to database in one bulk request. If -k- and -L- are greater than zero, they are written as well + * @param k (optional) is the database k parameter + * @param L (optional) is the database L parameter + * @throws ElasticException for communication problems with the server + */ + private void writeBasicInfo(int k, int L) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"index\": { \"_id\": \"name\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.databasename) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"owner\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.owner) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"description\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.description) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"major\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(Short.toString(info.major)) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"minor\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(Short.toString(info.minor)) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"settings\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(Integer.toString(info.settings)) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"readonly\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(SpecXmlUtils.encodeBoolean(info.readonly)) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"trackcallgraph\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(SpecXmlUtils.encodeBoolean(info.trackcallgraph)) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"layout\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(Integer.toString(info.layout_version)) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"datecolumn\" } }\n"); + String datecol = (info.dateColumnName == null) ? "Ingest Date" : info.dateColumnName; + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"").append(datecol).append("\" }\n"); + if (k > 0) { + buffer.append("{ \"index\": { \"_id\": \"k\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(Integer.toString(k)) + .append("\" }\n"); + buffer.append("{ \"index\": { \"_id\": \"L\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(Integer.toString(L)) + .append("\" }\n"); + } + connection.executeBulk('/' + repository + '_' + "configuration/_bulk", buffer.toString()); + writeExecutableCategories(); + writeFunctionTags(); + } + + /** + * Extract category information for this database into the DatabaseInformation object + * from the key/value map. Category information is stored as keys: execatcount, execat1, execat2, ... + * @param infoResult is the information object to populate + * @param keyValue is the map of key/value pairs + * @throws ElasticException for communication problems with the server + */ + private void readExecutableCategories(DatabaseInformation infoResult, + Map keyValue) throws ElasticException { + String countString = keyValue.get("execatcount"); + int count; + if (countString != null) { + count = Integer.parseInt(countString); + } + else { + count = 0; // key may not be present, assume 0 categories + } + if (count <= 0) { + infoResult.execats = null; + return; + } + infoResult.execats = new ArrayList<>(); + for (int i = 0; i < count; ++i) { + String key = "execat" + (i + 1); + String value = getCriticalValue(key, keyValue); + infoResult.execats.add(value); + } + } + + /** + * Extract function tag information for this database into the DatabaseInformation object + * from the key/value map. Tag information is stored as keys: functiontagcount, functiontag1, functiontag2, ... + * @param infoResult is the information object that will hold results + * @param keyValue is the keyword->value map for the database + * @throws ElasticException if a critical function tag key is missing + */ + private void readFunctionTags(DatabaseInformation infoResult, Map keyValue) + throws ElasticException { + String countString = keyValue.get("functiontagcount"); + int count; + if (countString != null) { + count = Integer.parseInt(countString); + } + else { + count = 0; // key may not be present, assume 0 tags + } + if (count <= 0) { + infoResult.functionTags = null; + return; + } + infoResult.functionTags = new ArrayList<>(); + for (int i = 0; i < count; ++i) { + String key = "functiontag" + (i + 1); + String value = getCriticalValue(key, keyValue); + infoResult.functionTags.add(value); + } + } + + /** + * Write out executable category information for this database as "keyvalue" documents + * @throws ElasticException for communication problems with the server + */ + private void writeExecutableCategories() throws ElasticException { + StringBuilder buffer = new StringBuilder(); + if (info.execats == null) { + buffer.append("{ \"index\": { \"_id\": \"execatcount\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"0\"}\n"); + } + else { + buffer.append("{ \"index\": { \"_id\": \"execatcount\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.execats.size()) + .append("\" }\n"); + for (int i = 0; i < info.execats.size(); ++i) { + buffer.append("{ \"index\": { \"_id\": \"execat").append(i + 1).append("\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.execats.get(i)) + .append("\" }\n"); + } + } + connection.executeBulk('/' + repository + '_' + "configuration/_bulk", buffer.toString()); + } + + /** + * Write out function tag information for this database as "keyvalue" documents + * @throws ElasticException for communication problems with the server + */ + private void writeFunctionTags() throws ElasticException { + StringBuilder buffer = new StringBuilder(); + if (info.functionTags == null) { + buffer.append("{ \"index\": { \"_id\": \"functiontagcount\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"0\"}\n"); + } + else { + buffer.append("{ \"index\": { \"_id\": \"functiontagcount\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.functionTags.size()) + .append("\" }\n"); + for (int i = 0; i < info.functionTags.size(); ++i) { + buffer.append("{ \"index\": { \"_id\": \"functiontag") + .append(i + 1) + .append("\" } }\n"); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.functionTags.get(i)) + .append("\" }\n"); + } + } + connection.executeBulk('/' + repository + '_' + "configuration/_bulk", buffer.toString()); + } + + /** + * Read all of the basic database configuration information from the "keyvalue" documents + * @param config is the Configuration object to fill in with the info + * @throws ElasticException for communication problems with the server + * @throws NoDatabaseException if the specific database does not exist on the server + */ + private void readBasicInfo(Configuration config) throws ElasticException, NoDatabaseException { + Map keyValue = readKeyValues(); + config.info.databasename = getCriticalValue("name", keyValue); + config.info.owner = getCriticalValue("owner", keyValue); + config.info.description = getCriticalValue("description", keyValue); + config.info.major = (short) SpecXmlUtils.decodeInt(getCriticalValue("major", keyValue)); + config.info.minor = (short) SpecXmlUtils.decodeInt(getCriticalValue("minor", keyValue)); + config.info.settings = SpecXmlUtils.decodeInt(getCriticalValue("settings", keyValue)); + config.info.readonly = SpecXmlUtils.decodeBoolean(getCriticalValue("readonly", keyValue)); + config.info.trackcallgraph = + SpecXmlUtils.decodeBoolean(getCriticalValue("trackcallgraph", keyValue)); + config.info.layout_version = SpecXmlUtils.decodeInt(getCriticalValue("layout", keyValue)); + config.info.dateColumnName = getCriticalValue("datecolumn", keyValue); + if (config.info.dateColumnName.equals("Ingest Date")) { + // name + config.info.dateColumnName = null; // Don't bother holding it + } + config.k = SpecXmlUtils.decodeInt(getCriticalValue("k", keyValue)); + config.L = SpecXmlUtils.decodeInt(getCriticalValue("L", keyValue)); + + readExecutableCategories(config.info, keyValue); + readFunctionTags(config.info, keyValue); + } + + /** + * Change the password for specific user. This assumes the ElasticSearch user + * in the native realm. + * @param uName is the user name + * @param password is the character data for the new password + * @throws ElasticException if the change is not successful + */ + private void changePasswordInternal(String uName, char[] password) throws ElasticException + + { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"password\": \""); + buffer.append(JSONObject.escape(String.valueOf(password))); + buffer.append("\" }"); + StringBuilder path = new StringBuilder(); + path.append("/_security/user/_password"); // Ignore the username, change for "current" user + connection.executeRawStatement(ElasticConnection.POST, path.toString(), buffer.toString()); + } + + /** + * Initialize a new BSim repository. This does most of the work of reading + * configuration info and setting up the various factories. + * @param config is the Configuration object that is populated during initialization + * @throws ElasticException for communication problems with the server + * @throws NoDatabaseException if the specific database does not exist on the server + */ + private void initializeElastic(Configuration config) + throws ElasticException, NoDatabaseException { + connection = new ElasticConnection(baseURL, repository); + if (baseURL.startsWith("https")) { + connectionType = ConnectionType.SSL_Password_Authentication; + } + config.info = new DatabaseInformation(); + readBasicInfo(config); + + vectorFactory = new Base64VectorFactory(); + config.weightfactory = new WeightFactory(); + config.idflookup = new IDFLookup(); + JSONObject res; + try { + res = connection.executeURIOnly(ElasticConnection.GET, "vector/_settings"); + } + catch (ElasticException ex) { + if (ex.getMessage().contains("no such index")) { + throw new NoDatabaseException(baseURL); + } + throw ex; + } + JSONObject repo = (JSONObject) res.get(repository + "_vector"); + JSONObject settings = (JSONObject) repo.get("settings"); + JSONObject index = (JSONObject) settings.get("index"); + JSONObject analysis = (JSONObject) index.get("analysis"); + JSONObject tokenizer = (JSONObject) analysis.get("tokenizer"); + String tokenizerName = null; + for (Object obj : tokenizer.keySet()) { + String key = (String) obj; + if (key.startsWith("lsh_")) { + tokenizerName = key; + break; + } + } + if (tokenizerName == null) { + throw new ElasticException("Missing tokenizer configuration"); + } + JSONObject tokenizerSettings = (JSONObject) tokenizer.get(tokenizerName); + String idfWeights = (String) tokenizerSettings.get(ElasticUtilities.LSH_WEIGHTS); + String[] split = idfWeights.split(" "); + double[] weightArray = new double[split.length]; + if (weightArray.length != config.weightfactory.getSize()) { + throw new ElasticException("weighttable has wrong number of rows"); + } + for (int i = 0; i < weightArray.length; ++i) { + weightArray[i] = Double.parseDouble(split[i]); + } + config.weightfactory.set(weightArray); + + String lookup = (String) tokenizerSettings.get(ElasticUtilities.IDF_CONFIG); + split = lookup.split(" "); + int[] lookupArray = new int[split.length]; + for (int i = 0; i < lookupArray.length; ++i) { + lookupArray[i] = Integer.parseInt(split[i]); + } + config.idflookup.set(lookupArray); + } + + @Override + public Status getStatus() { + return status; + } + + @Override + public ConnectionType getConnectionType() { + return connectionType; + } + + @Override + public String getUserName() { + if (userName != null) { + return userName; + } + return ClientUtil.getUserName(); + } + + @Override + public void setUserName(String userName) { + this.userName = userName; + } + + @Override + public LSHVectorFactory getLSHVectorFactory() { + return vectorFactory; + } + + @Override + public DatabaseInformation getInfo() { + return info; + } + + @Override + public int compareLayout() { + if (info.layout_version == ElasticDatabase.LAYOUT_VERSION) { + return 0; + } + return (info.layout_version < ElasticDatabase.LAYOUT_VERSION) ? -1 : 1; + } + + @Override + public BSimServerInfo getServerInfo() { + return serverInfo; + } + + @Override + public String getURLString() { + return baseURL + '/' + repository; + } + + @Override + public boolean initialize() { + if (initialized) { + return true; + } + try { + ClientUtil.getClientAuthenticator(); // Make sure an authenticator is installed + final Configuration config = new Configuration(); + initializeElastic(config); + info = config.info; + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + } + catch (ElasticException err) { + lastError = new Error(ErrorCategory.Initialization, + "Database error on initialization: " + err.getMessage()); + status = Status.Error; + return false; + } + catch (NoDatabaseException err) { + info = null; + lastError = new Error(ErrorCategory.Nodatabase, + "Database has not been created yet: " + err.getMessage()); + initialized = true; + status = Status.Ready; + return true; + } + status = Status.Ready; + initialized = true; + return true; + } + + @Override + public void close() { + if (connection != null) { + connection.close(); + connection = null; + } + status = Status.Unconnected; + initialized = false; + info = null; + } + + @Override + public Error getLastError() { + return lastError; + } + + @Override + public QueryResponseRecord query(BSimQuery query) { + if ((!isInitialized()) && (!(query instanceof CreateDatabase))) { + lastError = new Error(ErrorCategory.Nodatabase, "The database does not exist"); + return null; + } + lastError = null; + try { + query.buildResponseTemplate(); + if (query instanceof QueryNearest) { + fdbQueryNearest((QueryNearest) query); + } + else if (query instanceof QueryNearestVector) { + fdbQueryNearestVector((QueryNearestVector) query); + } + else if (query instanceof InsertRequest) { + fdbDatabaseInsert((InsertRequest) query); + } + else if (query instanceof QueryInfo) { + fdbDatabaseInfo((QueryInfo) query); + } + else if (query instanceof QueryName) { + fdbQueryName((QueryName) query); + } + else if (query instanceof QueryExeInfo) { + fdbQueryExeInfo((QueryExeInfo) query); + } + else if (query instanceof QueryExeCount) { + fdbQueryExeCount((QueryExeCount) query); + } + else if (query instanceof CreateDatabase) { + fdbDatabaseCreate((CreateDatabase) query); + } + else if (query instanceof QueryChildren) { + fdbQueryChildren((QueryChildren) query); + } + else if (query instanceof QueryDelete) { + fdbDelete((QueryDelete) query); + } + else if (query instanceof QueryUpdate) { + fdbUpdate((QueryUpdate) query); + } + else if (query instanceof QueryVectorId) { + fdbQueryVectorId((QueryVectorId) query); + } + else if (query instanceof QueryVectorMatch) { + fdbQueryVectorMatch((QueryVectorMatch) query); + } + else if (query instanceof QueryPair) { + fdbQueryPair((QueryPair) query); + } + else if (query instanceof InstallCategoryRequest) { + fdbInstallCategory((InstallCategoryRequest) query); + } + else if (query instanceof InstallTagRequest) { + fdbInstallTag((InstallTagRequest) query); + } + else if (query instanceof InstallMetadataRequest) { + fdbInstallMetadata((InstallMetadataRequest) query); + } + else if (query instanceof AdjustVectorIndex) { + fdbAdjustVectorIndex((AdjustVectorIndex) query); + } + else if (query instanceof PrewarmRequest) { + fdbPrewarm((PrewarmRequest) query); + } + else if (query instanceof PasswordChange) { + fdbPasswordChange((PasswordChange) query); + } + else { + lastError = new Error(ErrorCategory.Fatal, "Unknown query type"); + query.clearResponse(); + } + } + catch (DatabaseNonFatalException err) { + lastError = new Error(ErrorCategory.Nonfatal, + "Skipping -" + query.getName() + "- : " + err.getMessage()); + query.clearResponse(); + } + catch (LSHException err) { + lastError = new Error(ErrorCategory.Fatal, + "Fatal error during -" + query.getName() + "- : " + err.getMessage()); + query.clearResponse(); + } + catch (ElasticException err) { + lastError = new Error(ErrorCategory.Fatal, + "Elastic error during -" + query.getName() + "- : " + err.getMessage()); + query.clearResponse(); + } + return query.getResponse(); + } + + /** + * Create a new database + * @param config is the configuration information for the database + * @throws ElasticException for communication problems with the server + */ + private void generate(Configuration config) throws ElasticException { + config.info.layout_version = ElasticDatabase.LAYOUT_VERSION; + info = config.info; + vectorFactory = new Base64VectorFactory(); + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + + connection = new ElasticConnection(baseURL, repository); + + createConfigurationIndex(); + createExecutableIndex(); + createVectorIndex(config); + writeBasicInfo(config.k, config.L); + + status = Status.Ready; + initialized = true; + } + + /** + * Given the name of an executable library, its architecture, and a function name, + * return the id of the document describing this specific function. + * These 3 Strings are designed to uniquely identify a library function. + * @param exeName is the name of the executable + * @param funcName is the name of the function + * @param arch is the executable architecture + * @return the document id of the matching function + * @throws ElasticException if the function (the executable) doesn't exist + */ + public String recoverExternalFunctionId(String exeName, String funcName, String arch) + throws ElasticException { + String md5 = ExecutableRecord.calcLibraryMd5Placeholder(exeName, arch); + JSONObject row = queryMd5ExeMatch(md5); + if (row == null) { + throw new ElasticException( + "Could not resolve filter specifying executable: " + exeName); + } + String exeId = (String) row.get("_id"); + JSONArray descres = queryFuncNameMatch(exeId, funcName, 2); + if (1 != descres.size()) { + throw new ElasticException( + "Could not resolve filter specifying function: [" + exeName + "]" + funcName); + } + RowKeyElastic eKey = RowKeyElastic.parseExeIdString(exeId); + StringBuilder buffer = new StringBuilder(); + eKey.generateLibraryFunctionId(buffer, funcName); + return buffer.toString(); + } + + /** + * For every function currently in the manager, fill in its call-graph information. + * This involves querying the database for child information, adding the cross-link + * information (CallgraphEntry) between FunctionDescriptions, and possibly querying + * for new library executables and functions + * @param manager is the collection of functions to link + * @throws LSHException for problems updating the container + * @throws ElasticException for communication problems with the server + */ + private void queryCallgraph(DescriptionManager manager) throws LSHException, ElasticException { + if (!info.trackcallgraph) { + throw new LSHException("Database does not track callgraph"); + } + TreeMap funcmap = new TreeMap<>(); + manager.generateFunctionIdMap(funcmap); + for (ExecutableRecord exeRec : manager.getExecutableRecordSet()) { + if (exeRec.isLibrary()) { + continue; + } + List funclist = new ArrayList<>(); + Iterator iter = manager.listFunctions(exeRec); + while (iter.hasNext()) { // Build a static copy of the list of functions + funclist.add(iter.next()); + } + for (FunctionDescription element : funclist) { + fillinChildren(element, manager, funcmap); + } + } + } + + /** + * Entry point for the Elasticsearch version of QueryName command: + * Query for a specific executable and functions it contains + * @param query is command parameters + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding records to the response + */ + private void fdbQueryName(QueryName query) throws ElasticException, LSHException { + ResponseName response = query.nameresponse; + response.printselfsig = query.printselfsig; + response.printjustexe = query.printjustexe; + response.manage.setVersion(info.major, info.minor); + response.manage.setSettings(info.settings); + + ExecutableRecord erec = findSingleExecutable(query.spec, response.manage); + if (erec == null) { + response.uniqueexecutable = false; + return; + } + response.uniqueexecutable = true; + + queryByName(null, response.manage, erec, query.funcname, query.fillinSigs, query.maxfunc); + if (query.fillinCallgraph) { + queryCallgraph(response.manage); + } + } + + private String createExeFilter(String filterMd5, String filterExeName, String filterArch, + String filterCompilerName, boolean includeFakes) { + if (filterMd5 == null && filterExeName == null && filterArch == null && + filterCompilerName == null && includeFakes) { + return null; + } + boolean placedFirst = false; + StringBuilder buffer = new StringBuilder(); + buffer.append("\"filter\": [\n"); + if (filterMd5 != null) { + if (filterMd5.length() == 32) { // A complete md5 + buffer.append("{ \"term\" : { \"md5\" : \""); + buffer.append(filterMd5); + buffer.append("\" }}"); + } + else { // A partial md5 prefix + buffer.append("{ \"prefix\" : { \"md5\" : \""); + buffer.append(filterMd5); + buffer.append("\" }}"); + } + placedFirst = true; + } + if (filterExeName != null) { + if (placedFirst) { + buffer.append(",\n"); + } + buffer.append("{ \"wildcard\" : {"); + buffer.append(" \"name_exec\" : {"); + buffer.append(" \"value\" : \"*"); + buffer.append(JSONObject.escape(filterExeName)); + buffer.append("*\" }}}"); + placedFirst = true; + } + if (filterArch != null || filterCompilerName != null) { + if (placedFirst) { + buffer.append(",\n"); + } + buffer.append("{ \"script\": {"); + buffer.append(" \"script\": {"); + buffer.append(" \"inline\": \""); + if (filterArch == null) { // cname only + buffer.append("doc['name_compiler'].value == params.comp"); + } + else if (filterCompilerName == null) { // arch only + buffer.append("doc['architecture'].value == params.arch"); + } + else { // Both are provided + buffer.append( + "doc['name_compiler'].value == params.comp && doc['architecture'].value == params.arch"); + } + buffer.append("\","); + buffer.append(" \"params\": {"); + if (filterArch != null) { + buffer.append(" \"arch\": \"").append(filterArch); + if (filterCompilerName != null) { + buffer.append("\", "); + } + else { + buffer.append("\" "); + } + } + if (filterCompilerName != null) { + buffer.append(" \"comp\": \"").append(filterCompilerName).append("\" "); + } + buffer.append("}}}}"); + + } + buffer.append("]\n"); + if (!includeFakes) { + buffer.append(", \"must_not\" : "); + buffer.append("{ \"prefix\" : { \"md5\" : \"bbbbbbbbaaaaaaaa\" }}\n"); + } + return buffer.toString(); + } + + private void fdbQueryExeCount(QueryExeCount query) throws ElasticException { + ResponseExe response = query.exeresponse; + String filter = createExeFilter(query.filterMd5, query.filterExeName, query.filterArch, + query.filterCompilerName, query.includeFakes); + response.recordCount = countExecutables(filter); + } + + /** + * Queries the database for all executables matching the search criteria in the given + * {@link QueryExeInfo} object. Results are stored in the query info object + * + * @param query the query information + * @throws ElasticException if there is an error executing the query + * @throws LSHException if there is an error executing the query + */ + private void fdbQueryExeInfo(QueryExeInfo query) throws ElasticException, LSHException { + ResponseExe response = query.exeresponse; + String filter = createExeFilter(query.filterMd5, query.filterExeName, query.filterArch, + query.filterCompilerName, query.includeFakes); + queryExecutables(response.manage, response.records, query.limit, null, + query.sortColumn == ExeTableOrderColumn.MD5, filter); + response.recordCount = response.records.size(); + } + + /** + * Entry point for the Elasticsearch version of CreateDatabase command: + * Create a new database repository, with a specified configuration. + * @param query is command parameters + * @throws LSHException for problems loading the template + * @throws ElasticException for communication problems with the server + */ + private void fdbDatabaseCreate(CreateDatabase query) throws LSHException, ElasticException { + ResponseInfo response = query.inforesponse; + Configuration config = FunctionDatabase.loadConfigurationTemplate(query.config_template); + // Copy in any overriding fields in the query + if (query.info.databasename != null) { + config.info.databasename = query.info.databasename; + } + if (query.info.owner != null) { + config.info.owner = query.info.owner; + } + if (query.info.description != null) { + config.info.description = query.info.description; + } + if (!query.info.trackcallgraph) { + config.info.trackcallgraph = query.info.trackcallgraph; + } + if (query.info.functionTags != null) { + checkStrings(query.info.functionTags, "function tags", + FunctionTagBSimFilterType.MAX_TAG_COUNT); + config.info.functionTags = query.info.functionTags; + } + if (query.info.execats != null) { + checkStrings(query.info.execats, "categories", -1); + config.info.execats = query.info.execats; + } + generate(config); + response.info = config.info; + } + + private static void checkStrings(List list, String type, int limit) + throws LSHException { + if (limit > 0 && list.size() > limit) { + throw new LSHException("Too many " + type + " specified (limit=" + + FunctionTagBSimFilterType.MAX_TAG_COUNT + "): " + list.size()); + } + Set names = new HashSet<>(); + for (String name : list) { + if (!CategoryRecord.enforceTypeCharacters(name)) { + throw new LSHException("Bad characters in one or more proposed " + type); + } + if (!names.add(name)) { + throw new LSHException("Duplicate " + type + " entry specified: " + name); + } + } + } + + /** + * Entry point for the InstallCategoryRequest command: + * Install a new executable category to be managed by the database + * @param query is command parameters + * @throws LSHException if the command is misconfigured + * @throws ElasticException for communication problems with the server + */ + private void fdbInstallCategory(InstallCategoryRequest query) + throws LSHException, ElasticException { + ResponseInfo response = query.installresponse; + if (!CategoryRecord.enforceTypeCharacters(query.type_name)) { + throw new LSHException("Bad characters in proposed category type"); + } + if (query.isdatecolumn) { + info.dateColumnName = query.type_name; + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"type\": \"keyvalue\", \"value\": \"") + .append(info.dateColumnName) + .append("\" }"); + connection.executeStatementNoResponse(ElasticConnection.PUT, "configuration/datecolumn", + buffer.toString()); + response.info = info; + return; + } + // Check for existing category + if (info.execats != null) { + for (String cat : info.execats) { + if (cat.equals(query.type_name)) { + throw new LSHException("Executable category already exists"); + } + } + } + if (info.execats == null) { + info.execats = new ArrayList<>(); + } + info.execats.add(query.type_name); + writeExecutableCategories(); + response.info = info; + } + + /** + * Entry point for the Elasticsearch version of InstallTagRequest command: + * Install a new function tag to be managed by this data + * @param query is command parameters + * @throws LSHException if the command is misconfigured + * @throws ElasticException for communication problems with the server + */ + private void fdbInstallTag(InstallTagRequest query) throws LSHException, ElasticException { + final ResponseInfo response = query.installresponse; + if (!CategoryRecord.enforceTypeCharacters(query.tag_name)) { + throw new LSHException("Bad characters in proposed function tag"); + } + // Check for existing tag + if (info.functionTags != null) { + if (info.functionTags.contains(query.tag_name)) { + throw new LSHException("Function tag already exists"); + } + } + if (info.functionTags == null) { + info.functionTags = new ArrayList<>(); + } + // There are only 32-bits of space in the function record reserved for storing the presence of tags + if (info.functionTags.size() >= FunctionTagBSimFilterType.MAX_TAG_COUNT) { + throw new LSHException( + "Cannot allocate new function tag: " + query.tag_name + " - Column space is full"); + } + info.functionTags.add(query.tag_name); + writeFunctionTags(); + response.info = info; + } + + /** + * Entry point for the Elasticsearch version of InstallMetadataRequest command: + * Change some of the global meta-data labels for the database. + * @param query is command parameters + * @throws ElasticException for communication problems with the server + */ + private void fdbInstallMetadata(InstallMetadataRequest query) throws ElasticException { + final ResponseInfo response = query.installresponse; + if (query.dbname != null) { + info.databasename = query.dbname; + } + if (query.owner != null) { + info.owner = query.owner; + } + if (query.description != null) { + info.description = query.description; + } + if (query.dbname != null || query.owner != null || query.description != null) { + writeBasicInfo(0, 0); + } + response.info = info; + } + + /** + * Entry point for the Elasticsearch version of AdjustVectorIndex command: + * Adjust database settings pertinent to the main vector index + * @param query is command parameters + * @throws ElasticException for communication problems with the server + */ + private void fdbAdjustVectorIndex(AdjustVectorIndex query) throws ElasticException { + final ResponseAdjustIndex response = query.adjustresponse; + response.success = false; + response.operationSupported = true; + int numReplicas = query.doRebuild ? 1 : 0; + int refreshRateInSecs = query.doRebuild ? 1 : -1; + adjustReplicaRefresh("meta", numReplicas, refreshRateInSecs); + adjustReplicaRefresh("vector", numReplicas, refreshRateInSecs); + adjustReplicaRefresh("executable", numReplicas, refreshRateInSecs); + response.success = true; + } + + /** + * Entry point for the Elasticsearch version of PrewarmRequest command: + * This interface currently does not support prewarm + * @param request is command parameters + */ + private void fdbPrewarm(PrewarmRequest request) { + final ResponsePrewarm response = request.prewarmresponse; + response.operationSupported = false; + } + + /** + * Entry point for the Elasticsearch version of InsertRequest command: + * Insert new functions and executables into the database. + * @param query is command parameters + * @throws LSHException if the command is misconfigured + * @throws ElasticException for communication problems with the server + * @throws DatabaseNonFatalException if everything is already inserted + */ + private void fdbDatabaseInsert(InsertRequest query) + throws LSHException, ElasticException, DatabaseNonFatalException { + if (info.readonly) { + throw new LSHException("Trying to insert on read-only database"); + } + if (FunctionDatabase.checkSettingsForInsert(query.manage, info)) { // Check if settings are valid and is this is first insert + info.major = query.manage.getMajorVersion(); + info.minor = query.manage.getMinorVersion(); + info.settings = query.manage.getSettings(); + writeBasicInfo(0, 0); // Save off the settings associated with this first insert + } + ResponseInsert response = query.insertresponse; + if ((query.repo_override != null) && (query.repo_override.length() != 0)) { + query.manage.overrideRepository(query.repo_override, query.path_override); + } + // Insert each executable in turn + boolean newExecutable = false; + for (ExecutableRecord erec : query.manage.getExecutableRecordSet()) { + if (erec.isLibrary()) { + insertLibrary(query.manage, erec); + } + else if (insertExe(query.manage, erec)) { + newExecutable = true; + } + } + if (!newExecutable) { + throw new DatabaseNonFatalException("Already inserted"); + } + response.numexe = query.manage.getExecutableRecordSet().size(); + response.numfunc = query.manage.numFunctions(); + } + + /** + * Entry point for the Elasticsearch version of QueryPair command: + * Query for pairs functions in the database, and compute the similarity + * and significance of their feature vectors + * @param query is command parameters + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding records to the response + */ + private void fdbQueryPair(QueryPair query) throws ElasticException, LSHException { + ResponsePair response = query.pairResponse; + + double aveSim = 0.0; + double aveSimSquare = 0.0; + double aveSig = 0.0; + double aveSigSquare = 0.0; + int pairCount = 0; + int missedExe = 0; + int missedFunc = 0; + int missedVector = 0; + + List aFuncList = new ArrayList<>(); + List bFuncList = new ArrayList<>(); + DescriptionManager resManage = new DescriptionManager(); + TreeMap nameMap = new TreeMap<>(); + for (PairInput pairInput : query.pairs) { + FunctionDescription funcA = null; + FunctionDescription funcB = null; + ExecutableRecord erec = findSingleExeWithMap(pairInput.execA, resManage, nameMap); + if (erec == null) { + missedExe += 1; + } + else { + funcA = queryByNameAddress(resManage, erec, pairInput.funcA.funcName, + pairInput.funcA.address, true); + if (funcA == null) { + missedFunc += 1; + } + } + + erec = findSingleExeWithMap(pairInput.execB, resManage, nameMap); + if (erec == null) { + missedExe += 1; + } + else { + funcB = queryByNameAddress(resManage, erec, pairInput.funcB.funcName, + pairInput.funcB.address, true); + if (funcB == null) { + missedFunc += 1; + } + } + aFuncList.add(funcA); + bFuncList.add(funcB); + } + + Iterator bIter = bFuncList.iterator(); + VectorCompare vectorData = new VectorCompare(); + for (FunctionDescription funcA : aFuncList) { + FunctionDescription funcB = bIter.next(); + if (funcA == null || funcB == null) { + continue; + } + SignatureRecord sigA = funcA.getSignatureRecord(); + if (sigA == null) { + missedVector += 1; + continue; + } + SignatureRecord sigB = funcB.getSignatureRecord(); + if (sigB == null) { + missedVector += 1; + continue; + } + double sim = sigA.getLSHVector().compare(sigB.getLSHVector(), vectorData); + double signif = vectorFactory.calculateSignificance(vectorData); + PairNote pairNote = new PairNote(funcA, funcB, sim, signif, vectorData.dotproduct, + vectorData.acount, vectorData.bcount, vectorData.intersectcount); + response.notes.add(pairNote); + pairCount += 1; + aveSim += sim; + aveSimSquare += sim * sim; + aveSig += signif; + aveSigSquare += signif * signif; + } + response.averageSim = aveSim / pairCount; + response.averageSig = aveSig / pairCount; + double simVariance = (aveSimSquare / pairCount) - response.averageSim * response.averageSim; + response.simStdDev = Math.sqrt(simVariance); + double sigVariance = (aveSigSquare / pairCount) - response.averageSig * response.averageSig; + response.sigStdDev = Math.sqrt(sigVariance); + response.scale = vectorFactory.getSignificanceScale(); + response.pairCount = pairCount; + response.missedExe = missedExe; + response.missedFunc = missedFunc; + response.missedVector = missedVector; + } + + /** + * Entry point for the Elasticsearch version of QueryNearest command: + * Query for functions that are similar to those in the request. + * @param query is command parameters + * @throws LSHException for problems adding new records to the response + * @throws ElasticException for communication problems with the server + */ + private void fdbQueryNearest(QueryNearest query) throws LSHException, ElasticException { + FunctionDatabase.checkSettingsForQuery(query.manage, info); + String filter = null; + if (query.bsimFilter != null) { + ExecutableRecord repexe = query.manage.getExecutableRecordSet().first(); + IDElasticResolution idres[] = new IDElasticResolution[query.bsimFilter.numAtoms()]; + for (int i = 0; i < idres.length; ++i) { + FilterAtom atom = query.bsimFilter.getAtom(i); + idres[i] = atom.type.generateIDElasticResolution(atom); + if (idres[i] != null) { + idres[i].resolve(this, repexe); + } + } + filter = ElasticEffects.createFilter(query.bsimFilter, idres); + } + final ResponseNearest response = query.nearresponse; + response.totalfunc = 0; + response.totalmatch = 0; + response.uniquematch = 0; + + final DescriptionManager descMgr = new DescriptionManager(); + final Iterator iter = query.manage.listAllFunctions(); + + queryFunctions(query, filter, response, descMgr, iter); + response.manage.transferSettings(query.manage); // Echo back the settings + } + + /** + * Entry point for the Elasticsearch version of QueryNearestVector command: + * Query for vectors that are similar to those in the request + * @param query is command parameters + * @throws ElasticException for communication problems with the server + * @throws LSHException if the command is misconfigured + */ + private void fdbQueryNearestVector(QueryNearestVector query) + throws ElasticException, LSHException { + FunctionDatabase.checkSettingsForQuery(query.manage, info); + final ResponseNearestVector response = query.nearresponse; + response.totalvec = 0; + response.totalmatch = 0; + response.uniquematch = 0; + + int vectormax = query.vectormax; + if (vectormax == 0) { + vectormax = MAX_VECTOR_OVERALL; // Really means a very big limit + } + + final Iterator iter = query.manage.listAllFunctions(); + while (iter.hasNext()) { + final FunctionDescription frec = iter.next(); + final SignatureRecord srec = frec.getSignatureRecord(); + if (srec == null) { + continue; + } + final LSHVector thevec = srec.getLSHVector(); + final double len2 = vectorFactory.getSelfSignificance(thevec); + if (len2 < query.signifthresh) { + continue; + } + + response.totalvec += 1; + final List resultset = new ArrayList<>(); + + queryNearestVector(resultset, thevec, query.thresh, query.signifthresh, vectormax); + if (resultset.isEmpty()) { + continue; + } + final SimilarityVectorResult simres = new SimilarityVectorResult(frec); + simres.addNotes(resultset); + response.totalmatch += simres.getTotalCount(); + if (simres.getTotalCount() == 1) { + response.uniquematch += 1; + } + response.result.add(simres); + } + } + + private void fdbQueryVectorId(QueryVectorId query) throws ElasticException { + List resultList = query.vectorIdResponse.vectorResults; + for (Long id : query.vectorIds) { + VectorResult vecRes = new VectorResult(); + vecRes.vectorid = id; + resultList.add(vecRes); + } + Iterator iter1 = resultList.iterator(); + Iterator iter2 = resultList.iterator(); + while (iter1.hasNext()) { + fetchVectors(iter1, iter2, 50); // Fetch vector associated with each vectorid + } + iter1 = resultList.iterator(); + iter2 = resultList.iterator(); + while (iter1.hasNext()) { + fetchVectorCounts(iter1, iter2, MAX_VECTORCOUNT_WINDOW); // Fetch hitcount of each vector + } + } + + private void fdbQueryVectorMatch(QueryVectorMatch query) throws ElasticException, LSHException { + String filter = null; + if (query.bsimFilter != null) { + ExecutableRecord repexe = null; // Needed for ExternalFunction filter + IDElasticResolution idres[] = new IDElasticResolution[query.bsimFilter.numAtoms()]; + for (int i = 0; i < idres.length; ++i) { + FilterAtom atom = query.bsimFilter.getAtom(i); + idres[i] = atom.type.generateIDElasticResolution(atom); + if (idres[i] != null) { + idres[i].resolve(this, repexe); + } + } + filter = ElasticEffects.createFilter(query.bsimFilter, idres); + } + List vectorList = new ArrayList<>(); + for (Long id : query.vectorIds) { + VectorResult vecRes = new VectorResult(); + vecRes.vectorid = id; + vectorList.add(vecRes); + } + Iterator iter1 = vectorList.iterator(); + Iterator iter2 = vectorList.iterator(); + while (iter1.hasNext()) { + fetchVectors(iter1, iter2, 50); // Fetch vector associated with each vectorid + } + iter1 = vectorList.iterator(); + iter2 = vectorList.iterator(); + while (iter1.hasNext()) { + fetchVectorCounts(iter1, iter2, MAX_VECTORCOUNT_WINDOW); // Fetch hitcount of each vector + } + int count = 0; + DescriptionManager manager = query.matchresponse.manage; + for (VectorResult vecResult : vectorList) { + if (count >= query.max) { + break; + } + SignatureRecord srec = manager.newSignature(vecResult.vec, vecResult.hitcount); + JSONArray descres; + descres = queryVectorIdMatch(vecResult.vectorid, filter, query.max - count); + if (descres == null) { + throw new ElasticException( + "Error querying vectorid: " + Long.toString(vecResult.vectorid)); + } + if (descres.size() == 0) { + if (filter != null) { + continue; // Filter may have eliminated all results + } + // Otherwise this is a sign of corruption in the database + throw new ElasticException( + "No functions matching vectorid: " + Long.toString(vecResult.vectorid)); + } + count += descres.size(); + convertDescriptionRows(null, descres, vecResult, manager, srec); + } + } + + /** + * Entry point for the Elasticsearch version of QueryDelete command: + * Delete specific executables from the database + * @param query is command parameters + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems building records + */ + private void fdbDelete(QueryDelete query) throws ElasticException, LSHException { + final ResponseDelete response = query.respdelete; + for (ExeSpecifier spec : query.exelist) { + DescriptionManager manager = new DescriptionManager(); + ExecutableRecord erec = null; + if (spec.exemd5.length() != 0) { + JSONObject row = queryMd5ExeMatch(spec.exemd5); + if (row != null) { + erec = makeExecutableRecord(manager, row); + } + } + else { + erec = querySingleExecutable(manager, spec.exename, spec.arch, spec.execompname); + } + if (erec == null) { + response.missedlist.add(spec); + continue; + } + ResponseDelete.DeleteResult delrec = new ResponseDelete.DeleteResult(); + delrec.md5 = erec.getMd5(); + delrec.name = erec.getNameExec(); + List funclist = new ArrayList<>(); + RowKeyElastic eKey = updateKey(manager, erec); + String exeId = eKey.generateExeIdString(); + queryAllFunc(funclist, erec, exeId, manager, 0); + Set table = IdHistogram.buildVectorIdHistogram(funclist.iterator()); + List deleteList = new ArrayList<>(); + Iterator iter1 = table.iterator(); + Iterator iter2 = table.iterator(); + while (iter1.hasNext()) { + decrementVectorCounters(deleteList, iter1, iter2, MAX_VECTORDELETE_WINDOW); + } + iter1 = deleteList.iterator(); + while (iter1.hasNext()) { + deleteRawVectors(iter1, MAX_VECTORDELETE_WINDOW); + } + + delrec.funccount = deleteExeDocuments(exeId); + response.reslist.add(delrec); + } + } + + /** + * Entry point for the Elasticsearch version of QueryUpdate command: + * Update meta-data about specific executables and functions within the database + * @param query is command parameters + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems grouping records + */ + private void fdbUpdate(QueryUpdate query) throws ElasticException, LSHException { + ResponseUpdate response = query.updateresponse; + for (ExecutableRecord erec : query.manage.getExecutableRecordSet()) { + int res = updateExecutable(query.manage, erec, response.badfunc); + if (res < 0) { + response.badexe.add(erec); + } + else { + if ((res & 1) != 0) { + response.exeupdate += 1; + } + response.funcupdate += res >> 1; + } + } + } + + /** + * Entry point for the Elasticsearch version of PasswordChange command. + * @param query is command parameters + * @throws LSHException if details of the request are malformed + */ + private void fdbPasswordChange(PasswordChange query) throws LSHException { + ResponsePassword response = query.passwordResponse; + if (query.username == null) { + throw new LSHException("Missing username for password change"); + } + if (query.newPassword == null || query.newPassword.length == 0) { + throw new LSHException("No password provided"); + } + response.changeSuccessful = true; // Response parameters assuming success + response.errorMessage = null; + try { + changePasswordInternal(query.username, query.newPassword); + } + catch (ElasticException ex) { + response.changeSuccessful = false; + response.errorMessage = ex.getMessage(); + } + query.clearPassword(); + } + + /** + * Given the document id for a specific function. Query for the document and + * produce the corresponding FunctionDescription + * @param manager is the container for the new FunctionDescription + * @param rowId is the document id of the function + * @return the new FunctionDescription + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding records to the container + */ + private FunctionDescription querySingleDescriptionId(DescriptionManager manager, String rowId) + throws ElasticException, LSHException { + StringBuilder buffer = new StringBuilder(); + buffer.append("{ \"query\": { \"ids\": { \"values\": [ \""); + buffer.append(rowId); + buffer.append("\" ] } } }"); + JSONObject resp = connection.executeStatement(ElasticConnection.GET, "executable/_search", + buffer.toString()); + JSONObject hits = (JSONObject) resp.get("hits"); + JSONObject totalRec = (JSONObject) hits.get("total"); + long total = (Long) totalRec.get("value"); + if (total == 0) { + throw new ElasticException("No function documents matching id=" + rowId); + } + JSONArray hitsArray = (JSONArray) hits.get("hits"); + JSONObject row = (JSONObject) hitsArray.get(0); + JSONObject source = (JSONObject) row.get("_source"); + JSONObject joinfield = (JSONObject) source.get("join_field"); + String exeId = (String) joinfield.get("parent"); + RowKeyElastic eKey = RowKeyElastic.parseExeIdString(exeId); + ExecutableRecord exeRec = manager.findExecutableByRow(eKey); + if (exeRec == null) { + List keyList = new ArrayList<>(); + keyList.add(eKey); + queryExecutableRecordById(manager, keyList.iterator(), keyList.iterator(), 2); + exeRec = manager.findExecutableByRow(eKey); + } + return convertDescriptionRow(row, exeRec, manager, null); + } + + /** + * Given a specific function, query the database for the document ids of its children + * @param funcRecord is the specific function + * @return the child document ids as an array of JSON strings + * @throws ElasticException for communication problems with the server + */ + private JSONArray queryCallgraphRows(FunctionDescription funcRecord) throws ElasticException { + StringBuilder buffer = new StringBuilder(); + buffer.append("executable/_doc/"); + RowKeyElastic eKey = (RowKeyElastic) funcRecord.getExecutableRecord().getRowId(); + eKey.generateFunctionId(buffer, funcRecord); + buffer.append("?routing="); + buffer.append(eKey.generateExeIdString()); + buffer.append("&_source_includes=childid"); + JSONObject resp = connection.executeURIOnly(ElasticConnection.GET, buffer.toString()); + JSONObject source = (JSONObject) resp.get("_source"); + JSONArray childid = (JSONArray) source.get("childid"); + return childid; + } + + /** + * Given a specific function, query for all of its child functions. + * Uses a RowKey->FunctionDescription map to cache functions and avoid + * querying for the same function multiple times + * @param funcRecord is the specified function + * @param manager is the container for new child FunctionDescriptions + * @param functionMap is the cache + * @throws ElasticException for communication problems with the server + * @throws LSHException for problems adding records to the container + */ + private void fillinChildren(FunctionDescription funcRecord, DescriptionManager manager, + Map functionMap) throws ElasticException, LSHException { + if (!info.trackcallgraph) { + throw new ElasticException( + "Elasticsearch database does not have callgraph information enabled"); + } + JSONArray callids = queryCallgraphRows(funcRecord); + if (callids == null) { + return; // field is not present, meaning children are not present + } + for (Object callid : callids) { + String funcId = (String) callid; + RowKeyElastic eKey = RowKeyElastic.parseFunctionId(funcId); + FunctionDescription fdesc = functionMap.get(eKey); + if (fdesc == null) { + fdesc = querySingleDescriptionId(manager, funcId); + functionMap.put(eKey, fdesc); + } + manager.makeCallgraphLink(funcRecord, fdesc, 0); + } + } + + /** + * Entry point for Elasticsearch version of the QueryChildren command: + * Query for the child functins of submitted functions + * @param query is command parameters + * @throws LSHException for problems adding records to the response + * @throws ElasticException for communication problems with the server + */ + private void fdbQueryChildren(QueryChildren query) throws LSHException, ElasticException { + if (!info.trackcallgraph) { + throw new LSHException("Database does not track callgraph"); + } + ResponseChildren response = query.childrenresponse; + ExecutableRecord exe = null; + + ExeSpecifier exeSpec = new ExeSpecifier(); + exeSpec.exemd5 = query.md5sum; + exeSpec.exename = query.name_exec; + exeSpec.arch = query.arch; + exeSpec.execompname = query.name_compiler; + + exe = findSingleExecutable(exeSpec, response.manage); + if (exe == null) { + throw new LSHException("Could not (uniquely) match executable"); + } + for (FunctionEntry entry : query.functionKeys) { + FunctionDescription func = + queryByNameAddress(response.manage, exe, entry.funcName, entry.address, true); + if (func == null) { + throw new LSHException("Could not find function: " + entry.funcName); + } + response.correspond.add(func); + } + + TreeMap funcmap = new TreeMap<>(); + response.manage.generateFunctionIdMap(funcmap); + for (FunctionDescription element : response.correspond) { + fillinChildren(element, response.manage, funcmap); + } + } + + /** + * Entry point for the Elasticsearch version of QueryInfo command: + * Query for basic information about a database + * @param query is command parameters + */ + private void fdbDatabaseInfo(QueryInfo query) { + final ResponseInfo response = query.inforesponse; + response.info = info; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticEffects.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticEffects.java new file mode 100755 index 0000000000..ab4344d87d --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticEffects.java @@ -0,0 +1,255 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +import java.util.*; +import java.util.Map.Entry; + +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.query.protocol.*; + +/** + * Container for collecting an elasticsearch query filter document from BSimFilter elements + * + */ +public class ElasticEffects { + private int argumentCount = 0; + private int filterMask = 0; // Each 1-bit represents a single function tag that needs to be matched + private int filterValue = 0; // With the filterMask, bits indicate whether an individual + // function tag should match as true (1) or false(0) + + // Stand-alone filter element based on indexed executable fields + private Map> standaloneFilter = new TreeMap>(); + + // Set of executable docvalues we need for the script portion of the filter + private Set docValues = new TreeSet(); + + // Set of parameters used by the script portion of filter + private Map params = new TreeMap(); + + private Map dateParams = new TreeMap(); + + // Collection of elasticsearch script string pieces, sorted by the FilterTemplate that created them + private Map> booleanElements = + new TreeMap>(); + + // Ids of child functions, matching function must call + private Set childIds = new TreeSet(); + + private Map funcParams = new TreeMap(); + + public String assignArgument() { + argumentCount += 1; + return "arg" + argumentCount; + } + + public void addFunctionFilter(int flag, boolean val) { + filterMask |= flag; // Check the specific bit + if (val) { + filterValue |= flag; // must be set to 1 + } + } + + public void addStandalone(BSimFilterType filter,String value) { + List list = standaloneFilter.get(filter); + if (list == null) { + list = new ArrayList(); + standaloneFilter.put(filter, list); + } + list.add(value); + } + + public void addScriptElement(BSimFilterType filter,String value) { + List list = booleanElements.get(filter); + if (list == null) { + list = new ArrayList(); + booleanElements.put(filter, list); + } + list.add(value); + } + + public void addDocValue(String val) { + docValues.add(val); + } + + public void addParam(String key,String val) { + params.put(key, val); + } + + public void addDateParam(String key,Date date) { + dateParams.put(key, date.getTime()); + } + + public void addFuncParam(String key,String val) { + funcParams.put(key, val); + } + + public void addChildId(String id) { + childIds.add(id); + } + + private void buildStandaloneFilters(StringBuilder buffer) { + boolean needComma = false; + for(Entry> entry : standaloneFilter.entrySet()) { + List termList = entry.getValue(); + + for(String term : termList) { + if (needComma) { + buffer.append(','); + } + else { + needComma = true; + } + buffer.append(term); + } + } + } + + private void buildParentScript(StringBuilder buffer) { + buffer.append("\"inline\": \""); + for(String val : docValues) { + buffer.append(val); + } + buffer.append("return "); + boolean needsAnd = false; + for(Entry> entry : booleanElements.entrySet()) { + BSimFilterType filter = entry.getKey(); + String val = filter.buildElasticCombinedClause(entry.getValue()); + if (needsAnd) { + buffer.append(" && "); + } + else { + needsAnd = true; + } + buffer.append(val); + } + buffer.append("\""); + if ((!params.isEmpty()) || (!dateParams.isEmpty())) { + buffer.append(", \"params\": {"); + boolean needsComma = false; + for(Entry entry : params.entrySet()) { + if (needsComma) { + buffer.append(", "); + } + else { + needsComma = true; + } + buffer.append('\"').append(entry.getKey()).append("\": \""); + buffer.append(entry.getValue()).append('\"'); + } + for(Entry entry : dateParams.entrySet()) { + if (needsComma) { + buffer.append(", "); + } + else { + needsComma = true; + } + buffer.append('\"').append(entry.getKey()).append("\": "); + buffer.append(entry.getValue()); // Emit value as a JSON long integer + } + buffer.append("} "); + } + } + + private void buildParentFilterDocument(StringBuilder buffer) { + buffer.append("\"filter\": { "); + buffer.append("\"has_parent\": { "); + buffer.append("\"parent_type\": \"exe\", "); + buffer.append("\"query\": { "); + buffer.append("\"bool\": { "); + boolean needsComma = false; + if (!standaloneFilter.isEmpty()) { + buildStandaloneFilters(buffer); + needsComma = true; + } + if (!booleanElements.isEmpty()) { + if (needsComma) { + buffer.append(','); + } + buffer.append("\"filter\": { "); + buffer.append("\"script\": { \"script\": { "); + buildParentScript(buffer); + buffer.append("} } }"); + } + buffer.append("} } } }"); + } + + private void buildFunctionScriptFilter(StringBuilder buffer) { + boolean needsAnd = false; + buffer.append("\"filter\": {"); + buffer.append("\"script\": { \"script\": { "); + buffer.append("\"inline\": \""); + if (filterMask != 0) { + buffer.append("int flags = (int)doc['flags'].value; "); + } + if (!childIds.isEmpty()) { + buffer.append("def childid = doc['childid']; "); + } + buffer.append("return "); + if (filterMask !=0) { + buffer.append("((flags & params.mask) == params.value)"); + needsAnd = true; + } + for(String id : childIds) { + if (needsAnd) { + buffer.append(" && "); + } + buffer.append(id); + needsAnd = true; + } + buffer.append("\", \"params\": { "); + boolean needsComma = false; + if (filterMask != 0) { + buffer.append("\"mask\": ").append(filterMask); + buffer.append(", \"value\": ").append(filterValue); + needsComma = true; + } + for(Entry entry : funcParams.entrySet()) { + if (needsComma) { + buffer.append(", "); + } + else { + needsComma = true; + } + buffer.append('\"').append(entry.getKey()).append("\": \""); + buffer.append(entry.getValue()).append('\"'); + } + buffer.append("} } } }"); + } + + public String buildFunctionFilter() { + StringBuilder buffer = new StringBuilder(); + if ((filterMask != 0) || (!childIds.isEmpty())) { + buffer.append(", "); + buildFunctionScriptFilter(buffer); + } + if ((!booleanElements.isEmpty()) || (!standaloneFilter.isEmpty())) { + buffer.append(", "); + buildParentFilterDocument(buffer); + } + return buffer.toString(); + } + + public static String createFilter(BSimFilter filter,IDElasticResolution[] idres) throws ElasticException { + ElasticEffects effects = new ElasticEffects(); + + for (int i = 0; i < filter.numAtoms(); ++i) { + FilterAtom atom = filter.getAtom(i); + atom.type.gatherElasticEffect(effects, atom, idres[i]); + } + return effects.buildFunctionFilter(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticException.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticException.java new file mode 100755 index 0000000000..5f6b9d1d20 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticException.java @@ -0,0 +1,31 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +public class ElasticException extends Exception { + + private static final long serialVersionUID = 1L; + + public ElasticException(String msg) { + super(msg); + } + + @Override + public String toString() { + return "ElasticException: " + getMessage(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticUtilities.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticUtilities.java new file mode 100755 index 0000000000..f1516847b1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/ElasticUtilities.java @@ -0,0 +1,25 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +public class ElasticUtilities { + // Tokenizer Settings key names + public static final String K_SETTING = "k_setting"; + public static final String L_SETTING = "l_setting"; + public static final String LSH_WEIGHTS = "lsh_weights"; + public static final String IDF_CONFIG = "idf_config"; + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/Handler.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/Handler.java new file mode 100644 index 0000000000..e93a6705d2 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/Handler.java @@ -0,0 +1,49 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +import java.io.IOException; +import java.net.*; + +/** + * Dummy stream handler, so we can create URL objects with protocol "elastic" + */ +public class Handler extends URLStreamHandler { + + private static String MY_PARENT_PACKAGE = "ghidra.features.bsim.query"; + private static String PROTOCOL_HANDLER_PKGS_PROPERTY = "java.protocol.handler.pkgs"; + + @Override + protected URLConnection openConnection(URL u) throws IOException { + throw new IOException("Trying to open connection with dummy handler"); + } + + public static void registerHandler() { + String pkgs = System.getProperty(PROTOCOL_HANDLER_PKGS_PROPERTY); + if (pkgs != null) { + if (pkgs.indexOf(MY_PARENT_PACKAGE) >= 0) { + return; // avoid multiple registrations + } + pkgs = pkgs + "|" + MY_PARENT_PACKAGE; + } + else { + pkgs = MY_PARENT_PACKAGE; + } + + System.setProperty(PROTOCOL_HANDLER_PKGS_PROPERTY, pkgs); + + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/IDElasticResolution.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/IDElasticResolution.java new file mode 100755 index 0000000000..c8961c9e5a --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/IDElasticResolution.java @@ -0,0 +1,40 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +import ghidra.features.bsim.query.description.ExecutableRecord; + +public abstract class IDElasticResolution { + public String idString; + + public abstract void resolve(ElasticDatabase database,ExecutableRecord exe) throws ElasticException; + + public static class ExternalFunction extends IDElasticResolution { + private String exeName; // Name of executable containing external function + private String funcName; // Name of external function + + public ExternalFunction(String exe,String func) { + exeName = exe; + funcName = func; + idString = null; + } + + public void resolve(ElasticDatabase database,ExecutableRecord exe) throws ElasticException { + if (idString == null) + idString = database.recoverExternalFunctionId(exeName, funcName, exe.getArchitecture()); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/RowKeyElastic.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/RowKeyElastic.java new file mode 100755 index 0000000000..99a07da2a6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/elastic/RowKeyElastic.java @@ -0,0 +1,286 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.elastic; + +import generic.hash.SimpleCRC32; +import ghidra.features.bsim.query.description.*; + +/** + * A "document id" that uniquely indexes documents, within the ElasticSearch database, + * that describe executables {@link ExecutableRecord} and functions {@link FunctionDescription} + * This plays the same role as the row id for executable and function rows in an SQL + * database. + */ +public class RowKeyElastic extends RowKey { + protected int valueA, valueB, valueC; // Raw key data - 3 * 32 = 96 bits + + private static int hexString2Int(String val,int start) { + int res = 0; + for(int i=0;i<8;++i) { + res <<= 4; + char c = val.charAt(i+start); + if (c <= '9') { + res += (c-'0'); + } + else { + res += (c-'a') + 10; + } + } + return res; + } + + /** + * Initialize a key from a 64-bit long value + * @param val is (least significant) 64-bits of the key + */ + public RowKeyElastic(long val) { + valueA = 0; // Most significant 32-bits are 0 + valueB = (int)(val >>> 32); + valueC = (int)val; + } + + /** + * Create 96-bit, given 3 32-bit integers + * @param a is most significant 32-bits + * @param b is middle 32-bits + * @param c is least significant 32-bits + */ + public RowKeyElastic(int a,int b,int c) { + valueA = a; + valueB = b; + valueC = c; + } + + /** + * Construct key from String representation of an md5 hash. + * The key is initialized from the last 96-bits of the hash + * @param md5 is the hash + */ + public RowKeyElastic(String md5) { + valueA = hexString2Int(md5,8); + valueB = hexString2Int(md5,16); + valueC = hexString2Int(md5,24); + } + + /** + * Key initialized to zero + */ + public RowKeyElastic() { + valueA = 0; + valueB = 0; + valueC = 0; + } + + @Override + public long getLong() { + long res = valueB; + res = (res << 32) | (valueC & 0xffffffffL); + return res; + } + + @Override + public boolean equals(Object obj) { + if (obj == null) { + return false; + } + RowKeyElastic o = (RowKeyElastic)obj; + return (valueA == o.valueA) && (valueB == o.valueB) && (valueC == o.valueC); + } + + @Override + public int hashCode() { + int res = valueA; + res = res * 113 + valueB; + res = res * 113 + valueC; + return res; + } + + @Override + public int compareTo(RowKey obj) { + RowKeyElastic o = (RowKeyElastic)obj; + if (valueA != o.valueA) { + long valA = valueA & 0xffffffffL; + long ovalA = o.valueA & 0xffffffffL; + return (valA < ovalA) ? -1 : 1; + } + if (valueB != o.valueB) { + long valB = valueB & 0xffffffffL; + long ovalB = o.valueB & 0xffffffffL; + return (valB < ovalB) ? -1 : 1; + } + if (valueC != o.valueC) { + long valC = valueC & 0xffffffffL; + long ovalC = o.valueC & 0xffffffffL; + return (valC < ovalC) ? -1 : 1; + } + return 0; + } + + /** + * Emit the key as a base64 string of 16-characters. + * Used to encode executable document ids + * @return the String encoding + */ + public String generateExeIdString() { + StringBuilder buf = new StringBuilder(); + int curInt = valueA; + int chunk; + chunk = (curInt >> 26) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 20) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 14) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 8) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 2) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = ((curInt & 3) << 4); + curInt = valueB; + chunk = chunk | curInt >>> 28; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 22) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 16) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 10) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 4) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = ((curInt & 0xf) << 2); + curInt = valueC; + chunk = chunk | curInt >>> 30; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 24) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 18) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 12) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = (curInt >> 6) & 0x3f; + buf.append(Base64Lite.encode[chunk]); + chunk = curInt & 0x3f; + buf.append(Base64Lite.encode[chunk]); + return buf.toString(); + } + + /** + * Generate an encoded document id from 64 bits of this key + additional bits + * derived from a name string. This encodes the document id of a library function given + * just the function Name and the RowKey (this) of the containing library executable. + * The final String encodes 80-bits of id in 14 characters. + * @param buffer is the StringBuilder to encode the id to + * @param funcName is a function name that is hashed into the final encoded id + */ + public void generateLibraryFunctionId(StringBuilder buffer,String funcName) { + int hi = valueB; + int lo = valueC; + lo &= 0xffff0000; + for(int i=0;i>> 24; + lo = SimpleCRC32.hashOneByte(lo, funcName.charAt(i)); + hi = SimpleCRC32.hashOneByte(hi, tmp); + } + long res = hi; + res <<= 32; + res |= lo & 0xffffffffL; + int extra = valueC & 0xffff; + buffer.append(Base64Lite.encode[(extra >> 10)& 0x3f]); + buffer.append(Base64Lite.encode[(extra >> 4)& 0x3f]); + buffer.append(Base64Lite.encode[extra & 0xf]); + Base64Lite.encodeLongBase64Padded(buffer, res); + } + + /** + * Generate an id string for a FunctionDescription. If the function is not from a library, + * just use the counter id already set for the function and emit it as a decimal string. + * If it is from a library, emit an id, 4 bytes of which is from the md5 placeholder hash of the library, + * the rest of the id is a base64 encoding of a hash generated from: + * the remainder of the md5 placeholder hash of the library + * the name of the function + * @param buffer holds the emitted id string + * @param func is the function being labeled + */ + public void generateFunctionId(StringBuilder buffer,FunctionDescription func) { + ExecutableRecord exeRec = func.getExecutableRecord(); + if (!exeRec.isLibrary()) { + buffer.append(func.getId().getLong()); + return; + } + generateLibraryFunctionId(buffer, func.getFunctionName()); + } + + /** + * Parse an encoded document id of an executable back into a key + * @param id is the encoded String + * @return the decoded RowKey + */ + public static RowKeyElastic parseExeIdString(String id) { + int valueA,valueB,valueC; + valueA = Base64Lite.decode[id.charAt(0)]; + valueA <<= 6; + valueA |= Base64Lite.decode[id.charAt(1)]; + valueA <<= 6; + valueA |= Base64Lite.decode[id.charAt(2)]; + valueA <<= 6; + valueA |= Base64Lite.decode[id.charAt(3)]; + valueA <<= 6; + valueA |= Base64Lite.decode[id.charAt(4)]; + valueB = Base64Lite.decode[id.charAt(5)]; + valueA = (valueA << 2) | (valueB>>4); + valueB <<= 6; + valueB |= Base64Lite.decode[id.charAt(6)]; + valueB <<= 6; + valueB |= Base64Lite.decode[id.charAt(7)]; + valueB <<= 6; + valueB |= Base64Lite.decode[id.charAt(8)]; + valueB <<= 6; + valueB |= Base64Lite.decode[id.charAt(9)]; + valueC = Base64Lite.decode[id.charAt(10)]; + valueB = (valueB << 4) | (valueC >> 2); + valueC <<= 6; + valueC |= Base64Lite.decode[id.charAt(11)]; + valueC <<= 6; + valueC |= Base64Lite.decode[id.charAt(12)]; + valueC <<= 6; + valueC |= Base64Lite.decode[id.charAt(13)]; + valueC <<= 6; + valueC |= Base64Lite.decode[id.charAt(14)]; + valueC <<= 6; + valueC |= Base64Lite.decode[id.charAt(15)]; + return new RowKeyElastic(valueA,valueB,valueC); + } + + /** + * Parse an encoded document id of a function back into a key + * This handles both the normal function form: 64-bits encoded as decimal and + * the library function form: 80-bits encoded in base64 + * @param val is the encoded String + * @return the decoded RowKey + */ + public static RowKeyElastic parseFunctionId(String val) { + if (val.length() != 14) { + return new RowKeyElastic(Long.parseLong(val)); + } + int extra = Base64Lite.decode[val.charAt(0)]; + extra <<= 6; + extra |= Base64Lite.decode[val.charAt(1)]; + extra <<= 6; + extra |= Base64Lite.decode[val.charAt(2)]; + long low = Base64Lite.decodeLongBase64(val.substring(3)); + return new RowKeyElastic(extra,(int)(low>>>32),(int)low); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/DatabaseInfo.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/DatabaseInfo.java new file mode 100755 index 0000000000..27c27c0d91 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/DatabaseInfo.java @@ -0,0 +1,65 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.features.bsim.query.description.DatabaseInformation; + +public class DatabaseInfo { + + private final String serverURL; + private final DatabaseInformation databaseInformation; + + public DatabaseInfo(String serverURL, DatabaseInformation databaseInformation) { + this.serverURL = serverURL; + this.databaseInformation = databaseInformation; + } + + public String getServerURL() { + return serverURL; + } + + public String getName() { + return databaseInformation.databasename; + } + + public String getOwner() { + return databaseInformation.owner; + } + + public String getDescription() { + return databaseInformation.description; + } + + public String getVersion() { + return Short.toString(databaseInformation.major) + "." + + Short.toString(databaseInformation.minor); + } + + public boolean isReadOnly() { + return databaseInformation.readonly; + } + + @Override + public String toString() { + // @formatter:off + return "Database: " + serverURL + + "\ntName: " + getName() + + "\n\tOwner: " + getOwner() + + "\n\tVersion: " + getVersion() + + "\n\tDescription: " + getDescription(); + // @formatter:on + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/DefaultSFQueryServiceFactory.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/DefaultSFQueryServiceFactory.java new file mode 100755 index 0000000000..cca702a859 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/DefaultSFQueryServiceFactory.java @@ -0,0 +1,27 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.program.model.listing.Program; + +public class DefaultSFQueryServiceFactory extends SFQueryServiceFactory { + + @Override + public SimilarFunctionQueryService createSFQueryService(Program program) { + return new SimilarFunctionQueryService(program); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/FunctionSymbolIterator.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/FunctionSymbolIterator.java new file mode 100755 index 0000000000..9ee3d2b3cf --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/FunctionSymbolIterator.java @@ -0,0 +1,55 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.listing.Function; + +import java.util.Iterator; + +/** + * Convert an iterator over FunctionSymbols into an iterator over the Functions + */ +public class FunctionSymbolIterator implements Iterator { + + private Iterator symiter; + + public FunctionSymbolIterator(Iterator iter) { + symiter = iter; + } + + @Override + public boolean hasNext() { + return symiter.hasNext(); + } + + @Override + public Function next() { + FunctionSymbol sym = symiter.next(); + if (sym == null) + return null; + Object obj = sym.getObject(); + if (obj==null) return null; + return (Function)obj; + } + + @Override + public void remove() { + // not functional + + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/QueryDatabaseException.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/QueryDatabaseException.java new file mode 100755 index 0000000000..43db72220a --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/QueryDatabaseException.java @@ -0,0 +1,31 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +public class QueryDatabaseException extends Exception { + + public QueryDatabaseException(String message) { + super(message); + } + + public QueryDatabaseException(String message, Exception e) { + super(message, e); + } + + public QueryDatabaseException(Exception e) { + super(e); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFOverviewInfo.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFOverviewInfo.java new file mode 100755 index 0000000000..f80c32ff02 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFOverviewInfo.java @@ -0,0 +1,111 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import java.util.Set; + +import ghidra.features.bsim.query.protocol.PreFilter; +import ghidra.features.bsim.query.protocol.QueryNearestVector; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.listing.Program; + +public class SFOverviewInfo { + + public static final int DEFAULT_QUERIES_PER_STAGE = 10; // Default number of separate function queries to make at one time + + private Set functions; + private Program program; + private QueryNearestVector queryNearestVector; + private PreFilter preFilter; + + /** + * Constructs an overview request with default parameters. + * @param functions required--a set of functions (at least one) for which an overview will be + * computed. All functions must be from the same program. + * @throws IllegalArgumentException if functions is null/empty or functions + * are from multiple programs. + */ + public SFOverviewInfo(Set functions) { + if (functions == null) + throw new IllegalArgumentException("Function list cannot be null"); + if (functions.isEmpty()) + throw new IllegalArgumentException("Function list cannot be empty"); + + this.functions = functions; + for (FunctionSymbol s : functions) { + if (program == null) { + program = s.getProgram(); + } + else if (program != s.getProgram()) { + throw new IllegalArgumentException( + "all function symbols are not from the same program"); + } + } + queryNearestVector = new QueryNearestVector(); + preFilter = new PreFilter(); + } + + /** + * @return the program from which all queried functions are from + */ + public Program getProgram() { + return program; + } + + public double getSimilarityThreshold() { + return queryNearestVector.thresh; + } + + public void setSimilarityThreshold(double similarityThreshold) { + queryNearestVector.thresh = similarityThreshold; + } + + public double getSignificanceThreshold() { + return queryNearestVector.signifthresh; + } + + public void setSignificanceThreshold(double significanceThreshold) { + queryNearestVector.signifthresh = significanceThreshold; + } + + public int getVectorMax() { + return queryNearestVector.vectormax; + } + + public void setVectorMax(int max) { + queryNearestVector.vectormax = max; + } + + public QueryNearestVector buildQueryNearestVector() { + return queryNearestVector; + } + + public Set getFunctions() { + return functions; + } + + public int getNumberOfStages(int queries_per_stage) { + if ((functions == null)||(functions.size() == 0)) + return 1; + if (queries_per_stage == 0) + queries_per_stage = DEFAULT_QUERIES_PER_STAGE; + return (functions.size() + (queries_per_stage-1)) / queries_per_stage; + } + + public PreFilter getPreFilter(){ + return preFilter; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryInfo.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryInfo.java new file mode 100755 index 0000000000..d0ce3771c0 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryInfo.java @@ -0,0 +1,218 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import java.util.*; + +import ghidra.features.bsim.query.protocol.*; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.listing.Program; + +/** + * A simple container object to hold information that is to be sent to a database server as + * part of a query to find functions that are similar to those given in the constructor of this + * class. For a list of configurable parameters, see the setter methods of this class. + */ +public class SFQueryInfo { + + /** + * The number of queries to make for the given set of functions. For example, if 100 functions + * are submitted and the number of stages is 10, then 10 queries will be made to the server, + * with 10 functions per request. + *

+ * This defaults to 1, which means to send all functions in one query. + */ + public static final int DEFAULT_QUERIES_PER_STAGE = 10; + + private Set functions; + private Program program; + + private QueryNearest queryNearest; + private BSimFilter bsimFilter; + private PreFilter preFilter; + + /** + * Constructs a query request with default parameters. + * @param functions required--a set of functions (at least one) for which similar functions + * will searched. All functions must be from the same program. + * @throws IllegalArgumentException if functions is null/empty or functions + * are from multiple programs. + */ + public SFQueryInfo(Set functions) { + if (functions == null) { + throw new IllegalArgumentException("Function list cannot be null"); + } + + if (functions.isEmpty()) { + throw new IllegalArgumentException("Function list cannot be empty"); + } + + this.functions = functions; + for (FunctionSymbol s : functions) { + if (program == null) { + program = s.getProgram(); + } + else if (program != s.getProgram()) { + throw new IllegalArgumentException( + "all function symbols are not from the same program"); + } + } + queryNearest = new QueryNearest(); + bsimFilter = new BSimFilter(); + preFilter = new PreFilter(); + } + + /** + * @return the program from which all queried functions are from + */ + public Program getProgram() { + return program; + } + + /** + * Gets the threshold under which a potential similar function will not be matched. This + * threshold is for how similar the potential function is. This is a value from 0.0 to 1.0. The + * default value is {@value QueryNearest#DEFAULT_SIMILARITY_THRESHOLD}. + * + * @return threshold under which a potential similar function will not be matched. + */ + public double getSimilarityThreshold() { + return queryNearest.thresh; + } + + /** + * @see #getSimilarityThreshold() + * @param similarityThreshold the new threshold + */ + public void setSimilarityThreshold(double similarityThreshold) { + queryNearest.thresh = similarityThreshold; + } + + /** + * Gets the threshold under which a potential similar function will not be matched. This + * threshold is for how significant the match is (for example, smaller function matches + * are less significant). Higher is more significant. There is no upper bound. The + * default value is {@value QueryNearest#DEFAULT_SIGNIFICANCE_THRESHOLD}. + * + * @return threshold under which a potential similar function will not be matched. + */ + public double getSignificanceThreshold() { + return queryNearest.signifthresh; + } + + /** + * @see #getSignificanceThreshold() + * @param significanceThreshold the new threshold + */ + public void setSignificanceThreshold(double significanceThreshold) { + queryNearest.signifthresh = significanceThreshold; + } + + /** + * The maximum number of similar functions to return for a given input function + * The default value is {@value QueryNearest#DEFAULT_MAX_MATCHES}. + * + * @return The maximum number of similar functions to return + */ + public int getMaximumResults() { + return queryNearest.max; + } + + /** + * @see #getMaximumResults() + * @param maximumResults the new maximum + */ + public void setMaximumResults(int maximumResults) { + queryNearest.max = maximumResults; + } + + public QueryNearest buildQueryNearest() { + if (bsimFilter.isEmpty()) { + queryNearest.bsimFilter = null; + } + else { + queryNearest.bsimFilter = bsimFilter; + } + return queryNearest; + } + + /** + * Returns the input functions for which matches will be searched. + * @return the input functions for which matches will be searched. + */ + public Set getFunctions() { + return functions; + } + + /** + * Sets the input functions for which matches will be searched. + * @param functions the input functions for which matches will be searched. + */ + public void setFunctions(Set functions) { + this.functions = functions; + } + + public BSimFilter getBsimFilter() { + return bsimFilter; + } + + public PreFilter getPreFilter() { + return preFilter; + } + + public Collection getFilterInfoStrings() { + List arrlist = new ArrayList(); + for (int i = 0; i < bsimFilter.numAtoms(); ++i) { + FilterAtom atom = bsimFilter.getAtom(i); + String str = atom.getInfoString(); + if (str != null) { + arrlist.add(str); + } + } + return arrlist; + } + + /** + * The number of queries to make for the given set of functions. For example, if 100 functions + * are submitted and the number of stages is 10, then 10 queries will be made to the server, + * with 10 functions per request. + *

+ * This defaults to 1, which means to send all functions in one query. + * @param queries_per_stage how many queries to initiate per stage + * + * @return the number of queries to make for the given set of functions. + */ + public int getNumberOfStages(int queries_per_stage) { + if ((functions == null) || (functions.size() == 0)) { + return 1; + } + if (queries_per_stage == 0) { + queries_per_stage = DEFAULT_QUERIES_PER_STAGE; + } + return (functions.size() + (queries_per_stage - 1)) / queries_per_stage; + } + + @Override + public String toString() { + // @formatter:off + return getClass().getSimpleName() + + "\n\tsimilarity: " + queryNearest.thresh + + "\n\tsignificance: " + queryNearest.signifthresh + + "\n\tmax results: " + queryNearest.max + + "\n\tfunction count: " + functions.size(); + // @formatter:on + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryResult.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryResult.java new file mode 100755 index 0000000000..a01295bfe6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryResult.java @@ -0,0 +1,59 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import java.util.List; + +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.protocol.ResponseNearest; +import ghidra.features.bsim.query.protocol.SimilarityResult; + +/** + * The result of a call to {@link SimilarFunctionQueryService#querySimilarFunctions(SFQueryInfo, SFResultsUpdateListener, ghidra.util.task.TaskMonitor)} + */ +public class SFQueryResult { + + private final SFQueryInfo info; + private List resultlist; + private final DatabaseInfo facadeDatabaseInfo; + + SFQueryResult(SFQueryInfo info, String serverURL, DatabaseInformation databaseInformation, + ResponseNearest response) { + this.info = info; + this.resultlist = response.result; + this.facadeDatabaseInfo = new DatabaseInfo(serverURL, databaseInformation); + } + + /** + * The original query used to get the results represented by this object. + * @return the original query used to get the results represented by this object. + */ + public SFQueryInfo getQuery() { + return info; + } + + /** + * Returns the function database information representing the database server. + * @return the function database information representing the database server. + */ + public DatabaseInfo getDatabaseInfo() { + return facadeDatabaseInfo; + } + + public List getSimilarityResults() { + return resultlist; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryServiceFactory.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryServiceFactory.java new file mode 100755 index 0000000000..0bb02f3c11 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFQueryServiceFactory.java @@ -0,0 +1,23 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.program.model.listing.Program; + +public abstract class SFQueryServiceFactory { + + public abstract SimilarFunctionQueryService createSFQueryService(Program program); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFResultsUpdateListener.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFResultsUpdateListener.java new file mode 100755 index 0000000000..c33d102cef --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SFResultsUpdateListener.java @@ -0,0 +1,49 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.features.bsim.query.protocol.QueryResponseRecord; + +/** + * A listener that will be called as incremental results arrive from database queries. + * The results given to this listener are always a subset of the complete results. + * @param the final result implementation class. + */ +public interface SFResultsUpdateListener { + +// /** +// * Status callback +// * @param message status message +// * @param type message type +// */ +// void updateStatus(String message, MessageType type); +// + /** + * Called as incremental results arrive from database queries. The results given to + * this listener are always a subset of the complete results--they are not comprehensive. + * Consumer should be able to safely cast response based upon the type of query being performed. + * + * @param partialResponse a partial result record with the recently received results. + */ + public void resultAdded(QueryResponseRecord partialResponse); + + /** + * Callback to supply the final accumulated result. + * @param result accumulated query result or null if a failure occured which prevented + * results from being returned. + */ + public void setFinalResult(R result); +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SimilarFunctionQueryService.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SimilarFunctionQueryService.java new file mode 100755 index 0000000000..062b7ee460 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/facade/SimilarFunctionQueryService.java @@ -0,0 +1,543 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import java.net.MalformedURLException; +import java.net.URL; +import java.util.Set; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.decompiler.DecompileException; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.FunctionDatabase.ConnectionType; +import ghidra.features.bsim.query.FunctionDatabase.Status; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.protocol.*; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.listing.Program; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +/** + * A simple class that allows the user to query a server for functions that match a given + * set of functions. + */ +public class SimilarFunctionQueryService implements AutoCloseable { + + private FunctionDatabase database = null; + + private Program program; // program associated with signature generator + private GenSignatures signatureGenerator; // Cache of signature information collected so far + private int numStages; // Number of stages to place query with, 0 means get from query + + public SimilarFunctionQueryService(Program program) { + this.program = program; + numStages = 0; + } + + // for testing--dependency injection + SimilarFunctionQueryService(Program program, FunctionDatabase database) { + this.program = program; + this.database = database; + numStages = 0; + } + + /** + * Given a list of functions to query, prepare the final QueryNearest object which will be marshalled to the + * server. This involves generating the signatures for each of the function and accumulating their + * FunctionDescriptions + * @param queryInfo is the high-level form of query, with the list of FunctionSymbols and other parameters + * @param monitor the task monitor + * @return the QueryNearest object ready for the queryStaged method + * @throws QueryDatabaseException if transferring functions fails + */ + public QueryNearest generateQueryNearest(SFQueryInfo queryInfo, TaskMonitor monitor) + throws QueryDatabaseException { + QueryNearest result = queryInfo.buildQueryNearest(); + doSignatureGeneration(queryInfo.getFunctions(), monitor); + FunctionSymbolIterator iter = + new FunctionSymbolIterator(queryInfo.getFunctions().iterator()); + try { + signatureGenerator.transferCachedFunctions(result.manage, iter, + queryInfo.getPreFilter()); + } + catch (LSHException e) { + throw new QueryDatabaseException(e.getMessage()); + } + return result; + } + + public QueryNearestVector generateQueryNearestVector(SFOverviewInfo overviewInfo, + TaskMonitor monitor) throws QueryDatabaseException { + QueryNearestVector result = overviewInfo.buildQueryNearestVector(); + doSignatureGeneration(overviewInfo.getFunctions(), monitor); + FunctionSymbolIterator iter = + new FunctionSymbolIterator(overviewInfo.getFunctions().iterator()); + try { + signatureGenerator.transferCachedFunctions(result.manage, iter, + overviewInfo.getPreFilter()); + } + catch (LSHException e) { + throw new QueryDatabaseException(e.getMessage()); + } + return result; + } + + /** + * Issue password change request to the server + * @param username to change + * @param newPassword is password data + * @return null if change was successful, or the error message + */ + @Deprecated + public String changePassword(String username, char[] newPassword) { + if (database == null || database.getStatus() != Status.Ready) { + return "Connection not established"; + } + PasswordChange passwordChange = new PasswordChange(); + passwordChange.username = username; + passwordChange.newPassword = newPassword; + ResponsePassword response = passwordChange.execute(database); + if (!response.changeSuccessful) { + return response.errorMessage; + } + return null; + } + + /** + * Query the given server with the parameters provider by queryInfo. + * + * @param queryInfo a query info object containing the settings for the query + * @param listener is the listener to be informed of the query status and incremental results + * coming back, may be null + * @param monitor the task monitor to use; can be null + * @return the result object containing the retrieved similar functions; null if the query + * was cancelled + * @throws QueryDatabaseException if the query execution fails + * @throws CancelledException if the query is cancelled by the user + */ + public SFQueryResult querySimilarFunctions(SFQueryInfo queryInfo, + SFResultsUpdateListener listener, TaskMonitor monitor) + throws QueryDatabaseException, CancelledException { + SFQueryResult result = null; + try { + if (database == null || database.getStatus() != Status.Ready) { + throw new QueryDatabaseException("Connection with database not established"); + } + if (monitor == null) { + monitor = TaskMonitor.DUMMY; + } + if (listener == null) { + listener = new NullListener<>(); + } + + // + // Perform the required initialization: + // -Initialize signature generator + // -Hash the functions + // -Create the query + // -Create the staging + // + + QueryNearest query = generateQueryNearest(queryInfo, monitor); + int localNumStages = numStages; + if (localNumStages == 0) { + int funcsPerStage = database.getQueriedFunctionsPerStage(); + localNumStages = queryInfo.getNumberOfStages(funcsPerStage); + } + StagingManager stagingManager = + createStagingManager(queryInfo.getFunctions().size(), localNumStages); + + // + // Perform the query + // + ResponseNearest response = + (ResponseNearest) doQuery(query, stagingManager, listener, monitor); + + // + // Create the results for our facade interface + // + result = + new SFQueryResult(queryInfo, database.getURLString(), database.getInfo(), response); + } + catch (LSHException e) { + throw new QueryDatabaseException(e.getMessage()); + } + finally { + listener.setFinalResult(result); + } + return result; + } + + /** + * Query the given server for similar function overview information + * @param overviewInfo is details of the overview query + * @param listener is the listener to be informed of the query status and incremental results + * coming back, may be null + * @param monitor the task monitor + * @return the ResponseNearestVector + * @throws QueryDatabaseException if the database connection cannot be established + * @throws CancelledException if the query is cancelled by the user + */ + public ResponseNearestVector overviewSimilarFunctions(SFOverviewInfo overviewInfo, + SFResultsUpdateListener listener, TaskMonitor monitor) + throws QueryDatabaseException, CancelledException { + ResponseNearestVector response = null; + try { + if (database == null || database.getStatus() != Status.Ready) { + throw new QueryDatabaseException("Connection to database not established"); + } + if (monitor == null) { + monitor = TaskMonitor.DUMMY; + } + if (listener == null) { + listener = new NullListener<>(); + } + + QueryNearestVector query = generateQueryNearestVector(overviewInfo, monitor); + int localNumStages = numStages; + if (localNumStages == 0) { + int funcsPerStage = database.getOverviewFunctionsPerStage(); + localNumStages = overviewInfo.getNumberOfStages(funcsPerStage); + } + StagingManager stagingManager = + createStagingManager(overviewInfo.getFunctions().size(), localNumStages); + + try { + response = + (ResponseNearestVector) doQuery(query, stagingManager, listener, monitor); + } + catch (LSHException e) { + throw new QueryDatabaseException(e.getMessage()); + } + } + finally { + listener.setFinalResult(response); + } + return response; + } + + /** + * A lower-level (more flexible) query of the database. The query is not staged. + * @param query is the raw query information + * @param stagingManager is how to split up the query, can be null + * @param listener is the listener to be informed of the query status and incremental results + * coming back, may be null + * @param monitor the task monitor + * @return the raw response record from the database + * @throws QueryDatabaseException if the database connection cannot be established + * @throws CancelledException if the query is cancelled by the user + */ + public QueryResponseRecord queryRaw(BSimQuery query, StagingManager stagingManager, + SFResultsUpdateListener listener, TaskMonitor monitor) + throws QueryDatabaseException, CancelledException { + QueryResponseRecord response = null; + try { + if (database == null || database.getStatus() != Status.Ready) { + throw new QueryDatabaseException("Connection to database not established"); + } + if (monitor == null) { + monitor = TaskMonitor.DUMMY; + } + if (listener == null) { + listener = new NullListener<>(); + } + if (stagingManager == null) { + stagingManager = new NullStaging(); + } + response = doQuery(query, stagingManager, listener, monitor); + } + catch (LSHException e) { + throw new QueryDatabaseException(e.getMessage()); + } + finally { + listener.setFinalResult(response); + } + return response; + } + + public void dispose() { + close(); + } + + @Override + public void close() { + if (database != null) { + database.close(); + database = null; + } + if (signatureGenerator != null) { + signatureGenerator.dispose(); + signatureGenerator = null; + } + } + + public void setNumberOfStages(int val) { + numStages = val; + } + + public void updateProgram(Program newProgram) { + if (this.program != newProgram) { + this.program = newProgram; + this.signatureGenerator = null; + } + } + + private QueryResponseRecord doQuery(BSimQuery query, StagingManager stagingManager, + SFResultsUpdateListener listener, TaskMonitor monitor) + throws LSHException, CancelledException { + + boolean haveMore = stagingManager.initialize(query); + query.buildResponseTemplate(); + + QueryResponseRecord globalResponse = query.getResponse(); + + monitor.setMessage("Querying database"); + monitor.initialize(stagingManager.getTotalSize()); + + while (haveMore) { + monitor.checkCancelled(); + + // Get the current staged form of the query + BSimQuery stagedQuery = stagingManager.getQuery(); + + QueryResponseRecord response = stagedQuery.execute(database); + if (response != null) { + + if (globalResponse != response) { + globalResponse.mergeResults(response); // Merge the staged response with the global response + } + + listener.resultAdded(response); + + haveMore = stagingManager.nextStage(); + if (haveMore) { + stagedQuery.clearResponse(); // Make space for next stage + } + monitor.setProgress(stagingManager.getQueriesMade()); + } + else { + throw new LSHException(database.getLastError().message); + } + } + + return globalResponse; + } + + /** + * Return the {@link BSimServerInfo server info object} for this database + * @return the server info object or null if not currently associated with + * a {@link FunctionDatabase}. + */ + public BSimServerInfo getServerInfo() { + if (database == null) { + return null; + } + return database.getServerInfo(); + } + + public Status getDatabaseStatus() { + if (database == null) { + return Status.Unconnected; + } + return database.getStatus(); + } + + public ConnectionType getDatabaseConnectionType() { + if (database == null) { + return ConnectionType.Unencrypted_No_Authentication; + } + return database.getConnectionType(); + } + + public DatabaseInformation getDatabaseInformation() { + if (database == null) { + return null; + } + return database.getInfo(); + } + + public String getUserName() { + if (database == null) { + return null; + } + return database.getUserName(); + } + + public LSHVectorFactory getLSHVectorFactory() { + if (database == null) { + return null; + } + return database.getLSHVectorFactory(); + } + + public FunctionDatabase.Error getLastError() { + if (database == null) { + return null; + } + return database.getLastError(); + } + + /** + * Returns a string explaining the database compatibility between this client and the server. + * + * @return a string explaining the compatibility, or null if it could not be determined + */ + public String getDatabaseCompatibility() { + if (database == null) { + return null; + } + DatabaseInformation info = database.getInfo(); + if (info == null) { + return null; + } + int compare = database.compareLayout(); + if (compare < 0) { + return "This client is incompatible with the earlier database format on the server"; + } + else if (compare > 0) { + return "The server is using a later database format than is supported by this client"; + } + return null; + } + + // NOTE: Method overriden for testing + protected FunctionDatabase createDatabase(String urlString) throws MalformedURLException { + URL url = BSimClientFactory.deriveBSimURL(urlString); + return BSimClientFactory.buildClient(url, false); + } + + public void initializeDatabase(String serverURLString) throws QueryDatabaseException { + if (isSameDatabase(serverURLString)) { + if (database.getStatus() == Status.Ready) { + return; // Trying to connect with server which is still ready + } + } + + if (database != null) { + database.close(); // Shutdown old connection (or erroneous connection) + database = null; + } + + try { + database = createDatabase(serverURLString); + } + catch (MalformedURLException e) { + throw new QueryDatabaseException("Bad database URL: " + e.getMessage()); + } + boolean success = database.initialize(); + if (!success) { + String errorMsg = ""; + if (database.getLastError() != null) { + errorMsg = database.getLastError().message; + } + + throw new QueryDatabaseException(errorMsg); + } + } + + private boolean isSameDatabase(String serverURLString) { + if (database == null) { + return false; + } + return database.getURLString().equals(serverURLString); + } + + private void doSignatureGeneration(Set functions, TaskMonitor monitor) + throws QueryDatabaseException { + if (functions.isEmpty()) { + return; + } + + if (signatureGenerator == null) { + signatureGenerator = createSignatureGenerator(); + } + try { + signatureGenerator.setVectorFactory(database.getLSHVectorFactory()); + } + catch (LSHException e1) { + throw new QueryDatabaseException(e1); + } + + monitor.setMessage("Hashing function signatures..."); + FunctionSymbolIterator iter = new FunctionSymbolIterator(functions.iterator()); + int count = functions.size(); + try { +// TODO: do this work in a loop so that one failure doesn't stop the entire process...the +// downside is losing parallelization + signatureGenerator.scanFunctions(iter, count, monitor); + } + catch (DecompileException e) { + throw new QueryDatabaseException(e); + } + } + + private GenSignatures createSignatureGenerator() throws QueryDatabaseException { + try { + GenSignatures newSignatureGenerator = new GenSignatures(false); + newSignatureGenerator.openProgram(program, null, null, null, null, null); + return newSignatureGenerator; + } + catch (LSHException e) { + throw new QueryDatabaseException("Unable to signature functions", e); + } + } + + private StagingManager createStagingManager(int numqueries, int stages) { + if (stages == 1) { + return new NullStaging(); + } + + int numberOfFunctionsPerQuery; + if (stages > numqueries) { + // when the number of stages is greater than the number of functions, lower the stage + // count to execute one function at a time (stages becomes numqueries) + numberOfFunctionsPerQuery = 1; + } + else { + numberOfFunctionsPerQuery = (int) Math.ceil(numqueries / stages); + } + + return new FunctionStaging(numberOfFunctionsPerQuery); + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + + /** + * A dumby listener that will be called as incremental results arrive from database queries. + * No action is tacken for all results + * @param the final result implementation class. + */ + private class NullListener implements SFResultsUpdateListener { + + @Override + public void resultAdded(QueryResponseRecord response) { + // no-op + } + +// @Override +// public void updateStatus(String message, MessageType type) { +// // no-op +// } + + @Override + public void setFinalResult(R result) { + // TODO Auto-generated method stub + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/BSimH2FileDBConnectionManager.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/BSimH2FileDBConnectionManager.java new file mode 100644 index 0000000000..a5d13af20f --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/BSimH2FileDBConnectionManager.java @@ -0,0 +1,275 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.file; + +import java.io.File; +import java.net.URL; +import java.sql.Connection; +import java.sql.SQLException; +import java.util.*; + +import org.apache.commons.dbcp2.BasicDataSource; +import org.h2.tools.DeleteDbFiles; + +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.FunctionDatabase.ConnectionType; +import ghidra.features.bsim.query.FunctionDatabase.Status; + +public class BSimH2FileDBConnectionManager { + + private static final String DRIVER_CLASS_NAME = "org.h2.Driver"; + private static final int CONN_POOL_SIZE = 1; + private static final int CONN_POOL_MAX_IDLE = 2; + + /** + * Data source map keyed by absolute DB file path + */ + private static HashMap dataSourceMap = new HashMap<>(); + + /** + * Get all H2 File DB data sorces which exist in the JVM. + * @return all H2 File DB data sorces + */ + public static Collection getAllDataSources() { + // Create copy to avoid potential concurrent modification + return Collections.unmodifiableCollection(new ArrayList<>(dataSourceMap.values())); + } + + /** + * Get an existing or new H2 File DB data source for the specified H2 File + * specified by {@code fileServerInfo}. + * @param fileServerInfo H2 File DB info + * @return new or existing H2 File DB data source + * @throws IllegalArgumentException if {@code fileServerInfo} does not specify an + * H2 File DB type. + */ + public static BSimH2FileDataSource getDataSource(BSimServerInfo fileServerInfo) { + if (fileServerInfo.getDBType() != DBType.file) { + throw new IllegalArgumentException("expected file info"); + } + return dataSourceMap.computeIfAbsent(fileServerInfo, + info -> new BSimH2FileDataSource(info)); + } + + @Deprecated + public static BSimH2FileDataSource getDataSource(URL h2FileUrl) { + return getDataSource(new BSimServerInfo(h2FileUrl)); + } + + /** + * Get the existing H2 File DB data source for the specified BSim DB server info. + * This may return null if the H2 File DB exists but a + * {@link #getDataSource(BSimServerInfo) data source} + * has not yet been established within the running JVM. + * @param serverInfo BSim DB server info + * @return existing H2 File data source or null if server info does not correspond to an + * H2 File or has not be established as an H2 File data source. + */ + public static BSimH2FileDataSource getDataSourceIfExists(BSimServerInfo serverInfo) { + return dataSourceMap.get(serverInfo); + } + + private static synchronized void remove(BSimServerInfo serverInfo, boolean force) { + BSimH2FileDataSource ds = dataSourceMap.get(serverInfo); + if (ds == null) { + return; + } + int n = ds.bds.getNumActive(); + if (n != 0) { + System.out + .println("Unable to remove data source which has " + n + " active connections"); + if (!force) { + return; + } + } + ds.close(); + dataSourceMap.remove(serverInfo); + BSimVectorStoreManager.remove(serverInfo); + } + + /** + * {@link BSimH2FileDataSource} provides a pooled DB data source for a specific H2 File DB. + */ + public static class BSimH2FileDataSource implements BSimJDBCDataSource { + + private final BSimServerInfo serverInfo; + + private boolean successfulConnection = false; + + private BasicDataSource bds = new BasicDataSource(); + private BSimDBConnectTaskCoordinator taskCoordinator; + + private BSimH2FileDataSource(BSimServerInfo serverInfo) { + this.serverInfo = serverInfo; + this.taskCoordinator = new BSimDBConnectTaskCoordinator(serverInfo); + } + + @Override + public BSimServerInfo getServerInfo() { + return serverInfo; + } + + public void dispose() { + BSimH2FileDBConnectionManager.remove(serverInfo, true); + } + + /** + * Delete the database files associated with this H2 File DB. When complete + * this data source will no longer be valid and should no tbe used. + */ + public void delete() { + dispose(); + + File dbf = new File(serverInfo.getDBName()); + + // TODO: Should we check for lock on database - could be another process + + String name = dbf.getName(); + int ix = name.lastIndexOf(BSimServerInfo.H2_FILE_EXTENSION); + if (ix > 0) { + name = name.substring(0, ix); + } + + DeleteDbFiles.execute(dbf.getParent(), name, true); + } + + /** + * Determine if the stored DB file exists. + * @return true if the stored DB file exists + */ + public boolean exists() { + File dbf = new File(serverInfo.getDBName()); + return dbf.isFile(); + } + + private void close() { + try { + bds.close(); + } + catch (SQLException e) { + // ignore + } + } + + @Override + public Status getStatus() { + if (bds.isClosed()) { + return Status.Unconnected; + } + if (successfulConnection) { + return Status.Ready; + } + return Status.Error; + } + + @Override + public int getActiveConnections() { + return bds.getNumActive(); + } + + private String getH2FileUrl() { + + // Remove H2 db file extension if present + String dbName = serverInfo.getDBName(); + int ix = dbName.lastIndexOf(BSimServerInfo.H2_FILE_EXTENSION); + if (ix > 0) { + dbName = dbName.substring(0, ix); + } + + // On Windows we must remove the leading separator before the drive letter + if (File.separatorChar == '\\' && dbName.length() > 3 && dbName.charAt(0) == '/' && + Character.isLetter(dbName.charAt(1)) && dbName.charAt(2) == ':') { + // Remove leading '/' before drive letter + dbName = dbName.substring(1); + } + + return "jdbc:h2:" + dbName; + } + + private void setDefaultProperties() { + + // Set database driver name + bds.setDriverClassName(DRIVER_CLASS_NAME); + + // Set database URL + // NOTE: keywords 'key' and 'value' are used by KeyValueTable as column names + bds.setUrl(getH2FileUrl() + + ";MODE=PostgreSQL;DATABASE_TO_LOWER=TRUE;DEFAULT_NULL_ORDERING=HIGH;NON_KEYWORDS=key,value"); + + // Set the connection pool size + bds.setInitialSize(CONN_POOL_SIZE); + + // Set maximum number of idle connections + bds.setMaxIdle(CONN_POOL_MAX_IDLE); + + // Validate connection borrowed from pool + //bds.setValidationQuery("SELECT 1"); + //bds.setTestOnBorrow(true); + + // bds.setLogAbandoned(true); + // bds.setAbandonedUsageTracking(true); + } + + /** + * Get a connection to the H2 file database. + * It is important to note that if the database does not exist and empty one will + * be created. The {@link #exists()} method should be used to check for the database + * existance prior to connecting the first time. + * @return database connection + * @throws SQLException if a database error occurs + */ + @Override + public synchronized Connection getConnection() throws SQLException { + + if (successfulConnection) { + return bds.getConnection(); + } + + setDefaultProperties(); + + return taskCoordinator.getConnection(() -> connect()); + } + + @Override + public ConnectionType getConnectionType() { + return ConnectionType.Unencrypted_No_Authentication; + } + + @Override + public boolean equals(Object obj) { + if (obj instanceof BSimH2FileDataSource ds) { + return bds.getUrl().equals(ds.bds.getUrl()); + } + return false; + } + + @Override + public int hashCode() { + return bds.getUrl().hashCode(); + } + + /** + * Establish H2 File DB {@link Connection} performing any required authentication. + * @throws SQLException if connection or authentication error occurs + */ + private Connection connect() throws SQLException { + Connection c = bds.getConnection(); + successfulConnection = true; + return c; + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/BSimVectorStoreManager.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/BSimVectorStoreManager.java new file mode 100644 index 0000000000..d9ad9ca09a --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/BSimVectorStoreManager.java @@ -0,0 +1,35 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.file; + +import java.util.HashMap; +import java.util.Map; + +import ghidra.features.bsim.query.BSimServerInfo; + +public class BSimVectorStoreManager { + + private static Map vectorStoreMap = new HashMap<>(); + + public static synchronized VectorStore getVectorStore(BSimServerInfo serverInfo) { + return vectorStoreMap.computeIfAbsent(serverInfo, info -> new VectorStore(info)); + } + + public static synchronized void remove(BSimServerInfo serverInfo) { + vectorStoreMap.remove(serverInfo); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/H2FileFunctionDatabase.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/H2FileFunctionDatabase.java new file mode 100644 index 0000000000..3343454bfd --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/H2FileFunctionDatabase.java @@ -0,0 +1,356 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.file; + +import java.net.URL; +import java.sql.*; +import java.util.*; + +import generic.concurrent.*; +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.VectorCompare; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.client.*; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.elastic.Base64VectorFactory; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager.BSimH2FileDataSource; +import ghidra.features.bsim.query.protocol.*; +import ghidra.util.task.TaskMonitor; + +public class H2FileFunctionDatabase extends AbstractSQLFunctionDatabase { + + public static final int OVERVIEW_FUNCS_PER_STAGE = 1024; + public static final int QUERY_FUNCS_PER_STAGE = 256; + private static final String H2_THREADPOOL_NAME = "H2_BSIM_THREADPOOL"; + + // Indicates the version of the db table configuration. This needs to be updated + // whenever changes are made to the table structure. + public static final int LAYOUT_VERSION = 1; + + private final BSimH2FileDataSource fileDs; + + private H2VectorTable vectorTable; + private VectorStore vectorStore; + + /** + * Constructor used to connect to an existing H2 file database + * @param bsimURL local file URL for H2 database + */ + public H2FileFunctionDatabase(URL bsimURL) { + this(BSimH2FileDBConnectionManager.getDataSource(bsimURL)); + } + + /** + * Constructor used to connect to an existing H2 file database + * @param serverInfo local file info for H2 database + */ + public H2FileFunctionDatabase(BSimServerInfo serverInfo) { + this(BSimH2FileDBConnectionManager.getDataSource(serverInfo)); + } + + /** + * Constructor used to connect to an existing H2 file database + * @param ds local file H2 data source + */ + private H2FileFunctionDatabase(BSimH2FileDataSource ds) { + super(ds, new Base64VectorFactory(), LAYOUT_VERSION); + fileDs = ds; + vectorStore = BSimVectorStoreManager.getVectorStore(getServerInfo()); + vectorTable = new H2VectorTable(vectorFactory, vectorStore); + } + + @Override + public void close() { + vectorTable.close(); + super.close(); + } + + @Override + protected void setConnectionOnTables(Connection db) { + vectorTable.setConnection(db); + super.setConnectionOnTables(db); + } + + @Override + protected Connection initConnection() throws SQLException { + if (getStatus() != Status.Ready && !fileDs.exists()) { + throw new SQLException( + "Database does not exist: " + fileDs.getServerInfo().getDBName()); + } + return super.initConnection(); + } + + @Override + protected void generateRawDatabase() throws SQLException { + BSimServerInfo serverInfo = fileDs.getServerInfo(); + if (fileDs.exists()) { + throw new SQLException("Database already exists: " + serverInfo.getDBName()); + } + try (Connection c = fileDs.getConnection()) { + // do nothing - should throw exception on error + } + } + + @Override + protected void createDatabase(Configuration config) throws SQLException { + try { + super.createDatabase(config); + + Connection db = initConnection(); + try (Statement st = db.createStatement()) { + vectorTable.create(st); + } + } + catch (final SQLException err) { + throw new SQLException("Could not create database: " + err.getMessage()); + } + } + + /** + * Create vector map which maps vector ID to {@link VectorStoreEntry} + * @return vector map + * @throws SQLException if error occurs while reading map data + */ + public Map readVectorMap() throws SQLException { + return vectorTable.readVectors(); + } + + @Override + protected int deleteVectors(long id, int countdiff) throws SQLException { + return vectorTable.deleteVector(id, countdiff); + } + + @Override + public QueryResponseRecord doQuery(BSimQuery query, Connection c) + throws SQLException, LSHException, DatabaseNonFatalException { + + if (query instanceof PrewarmRequest preWarmRequest) { + preWarmRequest.buildResponseTemplate(); + preWarmRequest.prewarmresponse.operationSupported = false; + } + else if (query instanceof PasswordChange passwordChangeRequest) { + passwordChangeRequest.buildResponseTemplate(); + passwordChangeRequest.passwordResponse.changeSuccessful = false; + passwordChangeRequest.passwordResponse.errorMessage = + "Unsupported operation for H2 backend"; + } + else if (query instanceof AdjustVectorIndex q) { + q.buildResponseTemplate(); + q.adjustresponse.operationSupported = false; + } + else { + return super.doQuery(query, c); + } + return query.getResponse(); + } + + @Override + protected VectorResult queryVectorId(long id) throws SQLException { + VectorResult rowres = vectorTable.queryVectorById(id); + if (rowres == null) { + throw new SQLException("Bad vector table rowid"); + } + return rowres; + } + + @Override + protected long storeSignatureRecord(SignatureRecord sigrec) throws SQLException { + // NOTE: ignore sigrec count - assume only one (1) + return vectorTable.updateVector(sigrec.getLSHVector(), 1); + } + + @Override + protected int queryNearestVector(List resultset, LSHVector vec, + double simthresh, double sigthresh, int max) throws SQLException { + VectorCompare comp; + List resultsToSort = new ArrayList<>(); + for (VectorStoreEntry entry : vectorStore) { + if (entry.selfSig() < sigthresh) { + continue; + } + vec.compare(entry.vec(), comp = new VectorCompare()); + double cosine = comp.dotproduct / (vec.getLength() * entry.vec().getLength()); + if (cosine <= simthresh) { + continue; + } + double sig = vectorFactory.calculateSignificance(comp); + if (sig <= sigthresh) { + continue; + } + resultsToSort + .add(new VectorResult(entry.id(), entry.count(), cosine, sig, entry.vec())); + } + resultsToSort.sort((r1, r2) -> Double.compare(r2.sim, r1.sim)); + int maxResults = Math.min(max, resultsToSort.size()); + for (int i = 0; i < maxResults; ++i) { + resultset.add(resultsToSort.get(i)); + } + return resultset.size(); + } + + @Override + protected void queryNearestVector(QueryNearestVector query) throws SQLException { + ResponseNearestVector response = query.nearresponse; + response.totalvec = 0; + response.totalmatch = 0; + response.uniquematch = 0; + + // Use really big vector limit if not specified + int vectormax = query.vectormax == 0 ? 2000000 : query.vectormax; + + List toQuery = getFuncsToQuery(query.manage, query.signifthresh); + response.totalvec = toQuery.size(); + + GThreadPool threadPool = GThreadPool.getSharedThreadPool(H2_THREADPOOL_NAME); + ConcurrentQBuilder evalBuilder = + new ConcurrentQBuilder<>(); + ConcurrentQ evalQ = + evalBuilder.setThreadPool(threadPool) + .setCollectResults(true) + .setMonitor(TaskMonitor.DUMMY) + .build((fd, m) -> { + List resultset = new ArrayList<>(); + queryNearestVector(resultset, fd.getSignatureRecord().getLSHVector(), + query.thresh, query.signifthresh, vectormax); + if (resultset.isEmpty()) { + return null; + } + SimilarityVectorResult simres = new SimilarityVectorResult(fd); + simres.addNotes(resultset); + return simres; + }); + + evalQ.add(toQuery); + try { + Collection> results = + evalQ.waitForResults(); + for (QResult result : results) { + SimilarityVectorResult simres = result.getResult(); + if (simres == null) { + continue; + } + response.totalmatch += simres.getTotalCount(); + if (simres.getTotalCount() == 1) { + response.uniquematch += 1; + } + response.result.add(simres); + } + } + catch (Exception e) { + return; //shouldn't happen + } + } + + @Override + public int queryFunctions(QueryNearest query, BSimSqlClause filter, ResponseNearest response, + DescriptionManager descMgr, Iterator iter) + throws SQLException, LSHException { + //TODO: is this what the method should return + //TODO: why is iter an argument? Why not just use query.manage.listAllFunctions() + + //build a QueryNearestVector from query to do the vector search in parallel + + QueryNearestVector qnv = new QueryNearestVector(); + qnv.manage.transferSettings(query.manage); + qnv.signifthresh = query.signifthresh; + qnv.thresh = query.thresh; + qnv.vectormax = query.vectormax; + qnv.buildResponseTemplate(); + + //currently the parallelized vector search does no de-duping + //so we compute the number of distinct vectors queried to return + Set distinctVecIds = new HashSet<>(); + + while (iter.hasNext()) { + FunctionDescription fd = iter.next(); + if (fd.getSignatureRecord() != null) { + distinctVecIds.add(fd.getSignatureRecord().getLSHVector().calcUniqueHash()); + qnv.manage.transferFunction(fd, true); + } + } + queryNearestVector(qnv); + ResponseNearestVector rnv = qnv.nearresponse; + + //seems fishy but agrees with AbstractSQLFunctionDatabase.queryFunctions + response.totalfunc = rnv.totalvec; + + for (SimilarityVectorResult simVecRes : rnv.result) { + FunctionDescription base = simVecRes.getBase(); + Iterator vecResults = simVecRes.iterator(); + SimilarityResult sim = new SimilarityResult(base); + sim.setTotalCount(simVecRes.getTotalCount()); + int funcsForVec = 0; + while (vecResults.hasNext()) { + if (funcsForVec >= query.max) { + break; + } + VectorResult dresult = vecResults.next(); + funcsForVec += + retrieveFuncDescFromVectors(dresult, descMgr, funcsForVec, query, filter, sim); + } + if (sim.size() == 0) { + continue; + } + response.totalmatch += 1; + if (sim.size() == 1) { + response.uniquematch += 1; + } + response.result.add(sim); + sim.transfer(response.manage, true); + } + return distinctVecIds.size(); + } + + private List getFuncsToQuery(DescriptionManager manager, double sigBound) { + Iterator iter = manager.listAllFunctions(); + List toQuery = new ArrayList<>(); + while (iter.hasNext()) { + FunctionDescription frec = iter.next(); + SignatureRecord srec = frec.getSignatureRecord(); + + if (srec == null) { + continue; + } + LSHVector thevec = srec.getLSHVector(); + double len2 = vectorFactory.getSelfSignificance(thevec); + + // Self significance should be bigger than the significance threshold + // (or its impossible our result can exceed the threshold) + if (len2 < sigBound) { + continue; + } + toQuery.add(frec); + } + return toQuery; + } + + @Override + public String formatBitAndSQL(String v1, String v2) { + return "BITAND(" + v1 + "," + v2 + ")"; + } + + @Override + public int getQueriedFunctionsPerStage() { + return QUERY_FUNCS_PER_STAGE; + } + + @Override + public int getOverviewFunctionsPerStage() { + return OVERVIEW_FUNCS_PER_STAGE; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/H2VectorTable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/H2VectorTable.java new file mode 100755 index 0000000000..480f9cd479 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/H2VectorTable.java @@ -0,0 +1,288 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.file; + +import java.io.*; +import java.sql.*; +import java.util.HashMap; +import java.util.Map; + +import generic.lsh.vector.LSHVector; +import ghidra.features.bsim.query.client.tables.CachedStatement; +import ghidra.features.bsim.query.client.tables.SQLComplexTable; +import ghidra.features.bsim.query.description.VectorResult; +import ghidra.features.bsim.query.elastic.Base64Lite; +import ghidra.features.bsim.query.elastic.Base64VectorFactory; + +public class H2VectorTable extends SQLComplexTable { + + // FIXME: refine column type and vector storage format (consider binary) + + public static final String TABLE_NAME = "h2_vectable"; + + private final Base64VectorFactory vectorFactory; + private final VectorStore vectorStore; // in-memory cache + + private final CachedStatement insert_stmt = new CachedStatement<>(); + private final CachedStatement select_by_rowid_stmt = new CachedStatement<>(); + private final CachedStatement select_id_by_hash_stmt = + new CachedStatement<>(); + private final CachedStatement update_by_hash_stmt = new CachedStatement<>(); + private final CachedStatement select_count_by_rowid_stmt = + new CachedStatement<>(); + private final CachedStatement update_by_rowid_stmt = new CachedStatement<>(); + + public H2VectorTable(Base64VectorFactory vectorFactory, VectorStore vectorStore) { + super(TABLE_NAME, "id"); + this.vectorFactory = vectorFactory; + this.vectorStore = vectorStore; + } + + @Override + public void close() { + insert_stmt.close(); + select_by_rowid_stmt.close(); + select_id_by_hash_stmt.close(); + update_by_hash_stmt.close(); + select_count_by_rowid_stmt.close(); + update_by_rowid_stmt.close(); + super.close(); + } + + @Override + public void create(Statement st) throws SQLException { + st.executeUpdate("CREATE TABLE " + TABLE_NAME + + "(id SERIAL PRIMARY KEY, count INTEGER, vec_hash BIGINT, vec CLOB)"); + st.executeUpdate("CREATE UNIQUE INDEX h2_vectable_index ON " + TABLE_NAME + " (vec_hash)"); + } + + @Override + public void drop(Statement st) throws SQLException { + vectorStore.invalidate(); + st.executeUpdate("DROP INDEX h2_vectable_index"); + super.drop(st); + } + + @Override + public long insert(Object... arguments) throws SQLException { + + if (arguments == null || arguments.length != 2) { + throw new IllegalArgumentException( + "Insert method for H2VectorTable accepts two arguments: count(int) and LSHVector"); + } + + int count = (int) arguments[0]; + LSHVector vec = (LSHVector) arguments[1]; + + PreparedStatement s = insert_stmt.prepareIfNeeded(() -> db.prepareStatement( + "INSERT INTO " + TABLE_NAME + " (count,vec_hash,vec) VALUES(?,?,?)", + Statement.RETURN_GENERATED_KEYS)); + + StringBuilder vecBuf = new StringBuilder(); + vec.saveBase64(vecBuf, Base64Lite.encode); + + s.setInt(1, count); + s.setLong(2, vec.calcUniqueHash()); + s.setString(3, vecBuf.toString()); + if (s.executeUpdate() != 1) { + throw new SQLException("Insert failed for vector table"); + } + long id; + try (ResultSet rs = s.getGeneratedKeys()) { + if (!rs.next()) { + throw new SQLException("Unable to obtain vector id for insert"); + } + id = rs.getLong(1); + } + vectorStore.update( + new VectorStoreEntry(id, vec, count, vectorFactory.getSelfSignificance(vec))); + return id; + } + + /** + * Read all vectors from table and generate an ID-based vector map + * @return vector map (ID->VectorStoreEntry) + * @throws SQLException if error occurs + */ + public Map readVectors() throws SQLException { + char[] vectorDecodeBuffer = Base64VectorFactory.allocateBuffer(); + HashMap map = new HashMap<>(); + try (Statement st = db.createStatement(); + ResultSet rs = st.executeQuery("SELECT id,count,vec FROM " + TABLE_NAME)) { + while (rs.next()) { + long id = rs.getLong(1); + int count = rs.getInt(2); + Reader r = new StringReader(rs.getString(3)); + LSHVector vec = vectorFactory.restoreVectorFromBase64(r, vectorDecodeBuffer); + VectorStoreEntry entry = + new VectorStoreEntry(id, vec, count, vectorFactory.getSelfSignificance(vec)); + map.put(id, entry); + } + } + catch (IOException e) { + throw new SQLException(e); // unexpected for StringReader + } + return map; + } + + /** + * Get vector details which correspond to specified vector ID + * @param id vector ID + * @return vector details + * @throws SQLException if error occurs + */ + public VectorResult queryVectorById(long id) throws SQLException { + + VectorStoreEntry entry = vectorStore.getVectorById(id); + if (entry != null) { + return new VectorResult(id, entry.count(), 0, 0, entry.vec()); + } + + PreparedStatement s = select_by_rowid_stmt.prepareIfNeeded( + () -> db.prepareStatement("SELECT id,count,vec FROM " + TABLE_NAME + " WHERE id = ?")); + s.setLong(1, id); + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { + throw new SQLException("Bad vector table rowid"); + } + char[] vectorDecodeBuffer = Base64VectorFactory.allocateBuffer(); + VectorResult rowres; + try { + rowres = new VectorResult(); + rowres.vectorid = rs.getLong(1); + rowres.hitcount = rs.getInt(2); + Reader r = new StringReader(rs.getString(3)); + rowres.vec = vectorFactory.restoreVectorFromBase64(r, vectorDecodeBuffer); + } + catch (final IOException e) { + throw new SQLException(e.getMessage()); // unexpected for StringReader + } + return rowres; + } + } + + /** + * Get vector count which correspond to specified vector ID + * @param id vector ID + * @return vector count + * @throws SQLException if error occurs + */ + private int queryVectorCountById(long id) throws SQLException { + PreparedStatement s = select_count_by_rowid_stmt.prepareIfNeeded( + () -> db.prepareStatement("SELECT count FROM " + TABLE_NAME + " WHERE id = ?")); + s.setLong(1, id); + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { + throw new SQLException("Bad vector table rowid"); + } + return rs.getInt(1); + } + } + + /** + * Update or insert vector table entry with the specified positive countDiff. + * @param vec vector + * @param countDiff positive vector count change + * @return vector ID which was updated or created + * @throws SQLException if an error occurs + */ + public long updateVector(LSHVector vec, int countDiff) throws SQLException { + + if (countDiff <= 0) { + throw new IllegalArgumentException("Invalid countDiff: " + countDiff); + } + + // TODO: it may be possible to optimize the technique employed here + + PreparedStatement s = update_by_hash_stmt.prepareIfNeeded(() -> db.prepareStatement( + "UPDATE " + TABLE_NAME + " SET count = count + ? WHERE vec_hash = ?")); + long vecHash = vec.calcUniqueHash(); + s.setInt(1, countDiff); + s.setLong(2, vecHash); + int rc = s.executeUpdate(); + if (rc == 0) { + return insert(countDiff, vec); + } + if (rc > 1) { + throw new SQLException("Unexpected updated row count: " + rc); + } + + s = select_id_by_hash_stmt.prepareIfNeeded(() -> db + .prepareStatement("SELECT id, count FROM " + TABLE_NAME + " WHERE vec_hash = ?")); + s.setLong(1, vecHash); + + long id; + int count; + try (ResultSet rs = s.executeQuery()) { + if (!rs.next()) { + throw new SQLException("Unknown vector hash"); + } + id = rs.getLong(1); + count = rs.getInt(2); + } + vectorStore.update( + new VectorStoreEntry(id, vec, count, vectorFactory.getSelfSignificance(vec))); + return id; + } + + /** + * Update vector table entry with the specified countDiff. Record will be removed + * if reduced vector count less-than-or-equal zero. + * @param id vector ID + * @param countDiff positive vector count reduction + * @return 0 if decrement short of 0, return 1 if record was removed, return + * -1 if there was a problem + * @throws SQLException if an error occurs + */ + public int deleteVector(long id, int countDiff) throws SQLException { + + if (countDiff <= 0) { + throw new IllegalArgumentException("Invalid countDiff: " + countDiff); + } + + // TODO: it may be possible to optimize the technique employed here + + PreparedStatement s = update_by_rowid_stmt.prepareIfNeeded(() -> db.prepareStatement( + "UPDATE " + TABLE_NAME + " SET count = count - ? WHERE id = ? AND count >= ?")); + s.setInt(1, countDiff); + s.setLong(2, id); + s.setInt(3, countDiff); // needed for comparison + int rc = s.executeUpdate(); + if (rc == 0) { + return -1; + } + if (rc > 1) { + throw new SQLException("Unexpected updated row count: " + rc); + } + + int count = queryVectorCountById(id); + if (count > 0) { + vectorStore.update(id, count); + return 0; + } + + delete(id); + return 1; + } + + @Override + public int delete(long id) throws SQLException { + int rc = super.delete(id); + vectorStore.delete(id); + return rc; + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/VectorStore.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/VectorStore.java new file mode 100644 index 0000000000..c3209072ec --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/VectorStore.java @@ -0,0 +1,109 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.file; + +import java.sql.SQLException; +import java.util.Iterator; +import java.util.Map; + +import org.apache.commons.collections4.iterators.EmptyIterator; + +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.util.Msg; + +public class VectorStore implements Iterable { + + private BSimServerInfo serverInfo; + private Map vectors = null; + + public VectorStore(BSimServerInfo serverInfo) { + if (serverInfo.getDBType() != DBType.file) { + throw new IllegalArgumentException("Unsupported DBType"); + } + this.serverInfo = serverInfo; + } + + private void init() { + if (vectors == null) { + try { + loadVectors(); + } + catch (SQLException e) { + // TODO: do we need different interface to properly convey error? + Msg.error(this, "Failed to load vectors for " + serverInfo + ": " + e.getMessage()); + } + } + } + + @SuppressWarnings("unchecked") + @Override + public synchronized Iterator iterator() { + init(); + if (vectors == null) { + return EmptyIterator.INSTANCE; + } + return vectors.values().iterator(); + } + + public synchronized VectorStoreEntry getVectorById(long id) { + init(); + if (vectors == null) { + return null; + } + return vectors.get(id); + } + + private void loadVectors() throws SQLException { + // NOTE: assume file DB (see constructor above) + try (H2FileFunctionDatabase fnDb = new H2FileFunctionDatabase(serverInfo)) { + if (!fnDb.initialize()) { + throw new SQLException(fnDb.getLastError().message); + } + vectors = fnDb.readVectorMap(); + } + } + + public synchronized void invalidate() { + vectors = null; + } + + public synchronized void update(VectorStoreEntry entry) { + if (vectors != null) { + vectors.put(entry.id(), entry); + } + } + + public synchronized void update(long id, int count) { + if (vectors == null) { + return; + } + VectorStoreEntry entry = vectors.get(id); + if (entry == null) { + invalidate(); + } + else { + vectors.put(id, new VectorStoreEntry(id, entry.vec(), count, entry.selfSig())); + } + } + + public synchronized void delete(long id) { + if (vectors != null) { + vectors.remove(id); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/VectorStoreEntry.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/VectorStoreEntry.java new file mode 100644 index 0000000000..899fa3ddb1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/file/VectorStoreEntry.java @@ -0,0 +1,29 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.file; + +import generic.lsh.vector.LSHVector; + +/** + * A record containing an {@link LSHVector} and a count of the number of functions in the + * database which share the vector + * @param id vector ID + * @param vec vector + * @param count count + * @param selfSig self-significance of vector (using database settings) + */ +public record VectorStoreEntry(long id, LSHVector vec, int count, double selfSig) { +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/BSimLaunchable.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/BSimLaunchable.java new file mode 100644 index 0000000000..9eedf9917d --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/BSimLaunchable.java @@ -0,0 +1,1059 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.ingest; + +import java.io.File; +import java.io.IOException; +import java.net.*; +import java.util.*; + +import org.xml.sax.SAXException; + +import ghidra.GhidraApplicationLayout; +import ghidra.GhidraLaunchable; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.protocol.ExeSpecifier; +import ghidra.features.bsim.query.protocol.QueryName; +import ghidra.framework.*; +import ghidra.framework.client.ClientUtil; +import ghidra.framework.client.HeadlessClientAuthenticator; +import ghidra.framework.data.DomainObjectAdapter; +import ghidra.framework.protocol.ghidra.GhidraURL; +import ghidra.net.SSLContextInitializer; +import ghidra.program.database.ProgramDB; +import ghidra.util.Msg; +import ghidra.util.SystemUtilities; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; +import utility.application.ApplicationLayout; + +public class BSimLaunchable implements GhidraLaunchable { + + private static final String BSIM_LOGGING_CONFIGURATION_FILE = "bsim.log4j.xml"; + + private static final int DEFAULT_LIST_EXE_LIMIT = 20; + + private static final Set COMMAND_SET = new HashSet<>(); + + private static String defineCommand(String command) { + COMMAND_SET.add(command); + return command; + } + + /** + * bsim commands + */ + private static final String COMMAND_CREATE_DATABASE = defineCommand("createdatabase"); + private static final String COMMAND_SET_METADATA = defineCommand("setmetadata"); + private static final String COMMAND_ADD_EXE_CATEGORY = defineCommand("addexecategory"); + private static final String COMMAND_ADD_FUNCTION_TAG = defineCommand("addfunctiontag"); + private static final String COMMAND_DROP_INDEX = defineCommand("dropindex"); + private static final String COMMAND_REBUILD_INDEX = defineCommand("rebuildindex"); + private static final String COMMAND_PREWARM = defineCommand("prewarm"); + private static final String COMMAND_GENERATE_SIGS = defineCommand("generatesigs"); + private static final String COMMAND_COMMIT_SIGS = defineCommand("commitsigs"); + private static final String COMMAND_GENERATE_UPDATES = defineCommand("generateupdates"); + private static final String COMMAND_COMMIT_UPDATES = defineCommand("commitupdates"); + private static final String COMMAND_DELETE = defineCommand("delete"); + private static final String COMMAND_LIST_FUNCTIONS = defineCommand("listfuncs"); + private static final String COMMAND_LIST_EXES = defineCommand("listexes"); + private static final String COMMAND_GET_EXE_COUNT = defineCommand("getexecount"); + private static final String COMMAND_DUMP_SIGS = defineCommand("dumpsigs"); + + private static Set COMMANDS_WITH_REPO_ACCESS = + Set.of(COMMAND_GENERATE_SIGS, COMMAND_GENERATE_UPDATES); + + /** + * Constants for the option parameters that can be set in the various commands. + */ + private static final String BSIM_URL_OPTION = "bsim="; + private static final String GHIDRA_URL_OPTION = "ghidra="; + private static final String NAME_OPTION = "name="; + private static final String OWNER_OPTION = "owner="; + private static final String DESCRIPTION_OPTION = "description="; + private static final String OVERRIDE_OPTION = "override="; + private static final String CONFIG_OPTION = "config="; + private static final String MD5_OPTION = "md5="; + private static final String MAX_FUNC_OPTION = "maxfunc="; + private static final String FILTER_OPTION = "filter="; + private static final String ARCH_OPTION = "arch="; + private static final String COMPILER_OPTION = "compiler="; + private static final String LIMIT_OPTION = "limit="; + private static final String SORT_COL_OPTION = "sortcol="; + + // Global options + private static final String USER_OPTION = "user="; + private static final String CERT_OPTION = "cert="; + + private static final Set GLOBAL_OPTIONS = Set.of(CERT_OPTION, USER_OPTION); + + // Boolean options + private static final String COMMIT_OPTION = "--commit"; + private static final String CATEGORY_DATE_OPTION = "--date"; + private static final String NO_CALLGRAPH_OPTION = "--nocallgraph"; + private static final String OVERWRITE_OPTION = "--overwrite"; + private static final String INCLUDE_LIBS_OPTION = "--includelibs"; + private static final String PRINT_SELF_SIGNIFICANCE_OPTION = "--printselfsig"; + private static final String CALL_GRAPH_OPTION = "--callgraph"; + private static final String PRINT_JUST_EXE_OPTION = "--printjustexe"; + + //@formatter:off + // Populate ALLOWED_OPTION_MAP for each command + private static final Set CREATE_DATABASE_OPTIONS = + Set.of(NAME_OPTION, OWNER_OPTION, DESCRIPTION_OPTION, NO_CALLGRAPH_OPTION); + private static final Set COMMIT_SIGS_OPTIONS = + Set.of(OVERRIDE_OPTION, GHIDRA_URL_OPTION); // url requires override param + private static final Set COMMIT_UPDATES_OPTIONS = Set.of(); + private static final Set DELETE_OPTIONS = + Set.of(MD5_OPTION, NAME_OPTION, ARCH_OPTION, COMPILER_OPTION); // one or more params required + private static final Set DROP_INDEX_OPTIONS = Set.of(); + private static final Set REBUILD_INDEX_OPTIONS = Set.of(); + private static final Set PREWARM_OPTIONS = Set.of(); + private static final Set SET_METADATA_OPTIONS = + Set.of(NAME_OPTION, OWNER_OPTION, DESCRIPTION_OPTION); + private static final Set ADD_EXE_CATEGORY_OPTIONS = Set.of(CATEGORY_DATE_OPTION); + private static final Set ADD_FUNCTION_TAG_OPTIONS = Set.of(); + private static final Set DUMP_SIGS_OPTIONS = + Set.of(MD5_OPTION, NAME_OPTION, ARCH_OPTION, COMPILER_OPTION); + private static final Set GENERATE_SIGS_OPTIONS = + Set.of(CONFIG_OPTION, BSIM_URL_OPTION, OVERWRITE_OPTION, COMMIT_OPTION); // config OR bsimURL + private static final Set GENERATE_UPDATES_OPTIONS = + Set.of(CONFIG_OPTION, BSIM_URL_OPTION, OVERWRITE_OPTION, COMMIT_OPTION); // config OR bsimURL + private static final Set LIST_FUNCTIONS_OPTIONS = + Set.of(MD5_OPTION, NAME_OPTION, ARCH_OPTION, COMPILER_OPTION, PRINT_SELF_SIGNIFICANCE_OPTION, CALL_GRAPH_OPTION, PRINT_JUST_EXE_OPTION, MAX_FUNC_OPTION); + private static final Set GET_EXECUTABLES_OPTIONS = + Set.of(MD5_OPTION, NAME_OPTION, ARCH_OPTION, COMPILER_OPTION, SORT_COL_OPTION, INCLUDE_LIBS_OPTION); + private static final Set GET_EXECUTABLES_COUNT_OPTIONS = + Set.of(MD5_OPTION, NAME_OPTION, ARCH_OPTION, COMPILER_OPTION, INCLUDE_LIBS_OPTION); + //@formatter:on + + private static final Map> ALLOWED_OPTION_MAP = new HashMap<>(); + static { + ALLOWED_OPTION_MAP.put(COMMAND_CREATE_DATABASE, CREATE_DATABASE_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_SET_METADATA, SET_METADATA_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_ADD_EXE_CATEGORY, ADD_EXE_CATEGORY_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_ADD_FUNCTION_TAG, ADD_FUNCTION_TAG_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_DROP_INDEX, DROP_INDEX_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_REBUILD_INDEX, REBUILD_INDEX_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_PREWARM, PREWARM_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_GENERATE_SIGS, GENERATE_SIGS_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_COMMIT_SIGS, COMMIT_SIGS_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_GENERATE_UPDATES, GENERATE_UPDATES_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_COMMIT_UPDATES, COMMIT_UPDATES_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_DELETE, DELETE_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_LIST_FUNCTIONS, LIST_FUNCTIONS_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_LIST_EXES, GET_EXECUTABLES_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_GET_EXE_COUNT, GET_EXECUTABLES_COUNT_OPTIONS); + ALLOWED_OPTION_MAP.put(COMMAND_DUMP_SIGS, DUMP_SIGS_OPTIONS); + } + + private URL ghidraURL; + private URL bsimURL; + + private String bsimURLOption; // Command-line option: bsim=.. + private String ghidraURLOption; // Command-line option: ghidra=.. + private String nameOption; // Command-line option: name=.. + private String ownerOption; // Command-line option: owner=.. + private String archOption; // Command-line option: arch=.. + private String compOption; // Command-line option: compiler=.. + private String descOption; // Command-line option: description= + private String filterOption; // Command-line option: filter= + private String configOption; // Command-line option: config=.. + private String md5Option; // Command-line option: md5=.. + private Integer maxFunc; // Command-line option: maxfunc=.. + private String certOption; // Command-line option: cert=.. + private String connectingUserName; // Command-line option: user=.. + private Integer limitOption; // Command-line option: limit=.. + private String sortColumn; // Command-line option: sortcol=.. + private boolean overrideOption; // Command-line option: override= + + private Set booleanOptions = new HashSet<>(); + + private GhidraApplicationLayout layout; + + /** + * Constructor for launching from the console + */ + public BSimLaunchable() { + } + + /** + * Clears the parameters that can be used for issuing commands. This is useful + * if you want to keep your established database connection and URL settings but + * wish to issue a new command. + */ + private void clearParams() { + ghidraURL = null; + bsimURL = null; + bsimURLOption = null; + ghidraURLOption = null; + connectingUserName = null; + nameOption = null; + ownerOption = null; + archOption = null; + compOption = null; + descOption = null; + filterOption = null; + configOption = null; + md5Option = null; + certOption = null; + limitOption = null; + sortColumn = null; + overrideOption = false; + booleanOptions.clear(); + } + + private BulkSignatures getBulkSignatures() + throws IllegalArgumentException, MalformedURLException { + BSimServerInfo serverInfo = null; + if (bsimURL != null) { + serverInfo = new BSimServerInfo(bsimURL); + } + return new BulkSignatures(serverInfo, connectingUserName); + } + + private void setupGhidraURL(String ghidraURLString) throws MalformedURLException { + + if (ghidraURLString == null) { + return; + } + + if (!GhidraURL.isGhidraURL(ghidraURLString)) { + throw new MalformedURLException("URL is not ghidra protocol: " + ghidraURLString); + } + ghidraURL = new URL(ghidraURLString); + if (!GhidraURL.isServerRepositoryURL(ghidraURL) && + !GhidraURL.isLocalProjectURL(ghidraURL)) { + throw new MalformedURLException("Invalid repository URL: " + ghidraURLString); + } + } + + /** + * Establish the URL for the ghidra server and/or the bsim server. At least one string must be non-null + * @param ghidraURLString is the URL string for the ghidra server + * @param bsimURLString is the URL string for the bsim server + * @throws MalformedURLException if there is a problem parsing the given URLs + * @throws IllegalArgumentException if unsupported URL use occurs + */ + private void setupURLs(String ghidraURLString, String bsimURLString) + throws MalformedURLException { + + if (ghidraURLString != null) { + setupGhidraURL(ghidraURLString); + } + + if (bsimURLString != null) { + bsimURL = BSimClientFactory.deriveBSimURL(bsimURLString); + if (ghidraURL == null) { + if ("file".equals(bsimURL.getProtocol())) { + throw new IllegalArgumentException( + "Unable to infer ghidra URL from BSim file DB URL"); + } + ghidraURLString = "ghidra://" + bsimURL.getHost() + bsimURL.getPath(); + setupGhidraURL(ghidraURLString); + } + } + else if (ghidraURLString != null) { + bsimURL = BSimClientFactory.deriveBSimURL(ghidraURLString); + } + } + + /** + * Read in any optional parameters, strip them from the parameter stream + * @param command command name + * @param params is the original array of command line parameters + * @param discard number of params already consumed + * @return an array of parameters with optional ones stripped + */ + private List readOptions(String command, String[] params, int discard) { + + Set allowedParams = ALLOWED_OPTION_MAP.get(command); + if (allowedParams == null) { + throw new IllegalArgumentException("Unsupported command: " + command); + } + + List subParams = new ArrayList(); + for (int i = discard; i < params.length; ++i) { + String option = params[i]; + + int ix = option.indexOf('='); + if (ix > 0) { + String checkOption = option.substring(0, ix + 1); // include '=' in option name + if (!GLOBAL_OPTIONS.contains(checkOption) && !allowedParams.contains(checkOption)) { + throw new IllegalArgumentException("Unsupported option use: " + checkOption); + } + } + else if (option.startsWith("--")) { + if (!GLOBAL_OPTIONS.contains(option) && !allowedParams.contains(option)) { + throw new IllegalArgumentException("Unsupported option use: " + option); + } + booleanOptions.add(option); + continue; + } + + if (option.startsWith(BSIM_URL_OPTION)) { + bsimURLOption = option.substring(BSIM_URL_OPTION.length()); + } + else if (option.startsWith(GHIDRA_URL_OPTION)) { + ghidraURLOption = option.substring(GHIDRA_URL_OPTION.length()); + } + else if (option.startsWith(NAME_OPTION)) { + nameOption = option.substring(NAME_OPTION.length()); + } + else if (option.startsWith(OWNER_OPTION)) { + ownerOption = option.substring(OWNER_OPTION.length()); + } + else if (option.startsWith(DESCRIPTION_OPTION)) { + descOption = option.substring(DESCRIPTION_OPTION.length()); + } + else if (option.startsWith(OVERRIDE_OPTION)) { + overrideOption = true; + ghidraURLOption = option.substring(OVERRIDE_OPTION.length()); + } + else if (option.startsWith(CONFIG_OPTION)) { + configOption = option.substring(CONFIG_OPTION.length()); + } + else if (option.startsWith(MD5_OPTION)) { + md5Option = option.substring(MD5_OPTION.length()); + } + else if (option.startsWith(MAX_FUNC_OPTION)) { + String val = option.substring(MAX_FUNC_OPTION.length()); + try { + maxFunc = Integer.valueOf(val); + if (maxFunc < 0) { + throw new IllegalArgumentException( + "Negative value not permitted for maxfunc"); + } + } + catch (NumberFormatException e) { + throw new IllegalArgumentException("Invalid decimal value for maxfunc: " + val); + } + } + else if (option.startsWith(FILTER_OPTION)) { + filterOption = option.substring(FILTER_OPTION.length()); + } + else if (option.startsWith(CERT_OPTION)) { // global option + certOption = option.substring(CERT_OPTION.length()); + } + else if (option.startsWith(USER_OPTION)) { // global option + connectingUserName = option.substring(USER_OPTION.length()); + } + else if (option.startsWith(ARCH_OPTION)) { + archOption = option.substring(ARCH_OPTION.length()); + } + else if (option.startsWith(COMPILER_OPTION)) { + compOption = option.substring(COMPILER_OPTION.length()); + } + else if (option.startsWith(LIMIT_OPTION)) { + String val = option.substring(LIMIT_OPTION.length()); + try { + limitOption = Integer.valueOf(val); + if (limitOption < 0) { + throw new IllegalArgumentException( + "Negative value not permitted for limit"); + } + } + catch (NumberFormatException e) { + throw new IllegalArgumentException("Invalid decimal value for limit: " + val); + } + } + else if (option.startsWith(SORT_COL_OPTION)) { + sortColumn = option.substring(SORT_COL_OPTION.length()); + } + else if (params[i].startsWith("--") || params[i].contains("=")) { + throw new IllegalArgumentException("Unknown option: " + params[i]); + } + else { + subParams.add(params[i]); + } + } + if (connectingUserName == null) { + connectingUserName = ClientUtil.getUserName(); + } + return subParams; + } + + private void checkRequiredParam(String[] params, int index, String name) { + if (params.length <= index) { + throw new IllegalArgumentException("Missing required parameter: " + name); + } + String p = params[index]; + if (p.startsWith("--") || p.contains("=")) { + throw new IllegalArgumentException( + "Missing required parameter (" + name + ") before specified option: " + p); + } + } + + /** + * Runs the command specified by the given set of params. + * + * @param params the parameters specifying the command + * @param monitor the task monitor + * @throws IllegalArgumentException if invalid params have been specified + * @throws Exception if there's an error during the operation + * @throws CancelledException if processing is cancelled + */ + public void run(String[] params, TaskMonitor monitor) throws Exception, CancelledException { + + checkRequiredParam(params, 0, "command"); + String command = params[0]; + if (!COMMAND_SET.contains(command)) { + throw new IllegalArgumentException("Missing or invalid command specified"); + } + + checkRequiredParam(params, 1, "URL"); + String urlstring = params[1]; + + monitor.setCancelEnabled(true); + + clearParams(); + List subParams = readOptions(command, params, 2); + + initializeApplication(command); + + if (COMMAND_CREATE_DATABASE.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doCreateDatabase(subParams); + } + else if (COMMAND_SET_METADATA.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doInstallMetadata(subParams); + } + else if (COMMAND_ADD_EXE_CATEGORY.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doInstallCategory(subParams); + } + else if (COMMAND_ADD_FUNCTION_TAG.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doInstallTags(subParams); + } + else if (COMMAND_DROP_INDEX.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doDropIndex(subParams); + } + else if (COMMAND_REBUILD_INDEX.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doRebuildIndex(subParams); + } + else if (COMMAND_PREWARM.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doPrewarm(subParams); + } + else if (COMMAND_GENERATE_SIGS.equals(command)) { + processSigAndUpdateOptions(urlstring); + doGenerateSigs(subParams, monitor); + } + else if (COMMAND_COMMIT_SIGS.equals(command)) { + if (overrideOption) { + setupURLs(ghidraURLOption, urlstring); + } + else { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + } + doCommitSigs(subParams, monitor); + } + else if (COMMAND_GENERATE_UPDATES.equals(command)) { + processSigAndUpdateOptions(urlstring); + doGenerateUpdates(subParams, monitor); + } + else if (COMMAND_COMMIT_UPDATES.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doCommitUpdates(subParams); + } + else if (COMMAND_DELETE.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doDeleteExecutable(subParams); + } + else if (COMMAND_LIST_FUNCTIONS.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doListFunctions(subParams); + } + else if (COMMAND_LIST_EXES.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doListExes(subParams); + } + else if (COMMAND_GET_EXE_COUNT.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doGetCount(subParams); + } + else if (COMMAND_DUMP_SIGS.equals(command)) { + bsimURL = BSimClientFactory.deriveBSimURL(urlstring); + doDumpSigs(subParams); + } + else { + throw new IllegalArgumentException("Unknown command: " + command); + } + } + + private void processSigAndUpdateOptions(String urlstring) throws MalformedURLException { + if (configOption != null) { + if (bsimURLOption != null) { + throw new IllegalArgumentException( + "bsim= and config= parameters may not both be present"); + } + setupGhidraURL(urlstring); + } + else if (bsimURLOption != null) { + setupURLs(urlstring, bsimURLOption); + } + else { + throw new IllegalArgumentException( + "Must specify either \"bsim=\" or \"config=\" option is required"); + } + } + + /** + * Runs the command specified by the given set of params. + * + * @param params the parameters specifying the command + * @throws Exception when initializing the application or executing the command + */ + public void run(String[] params) throws Exception { + run(params, TaskMonitor.DUMMY); + } + + /** + * Creates a new BSim database with a given set of properties. + * + * @param params the command-line parameters + * @throws IOException if there's an error establishing the database connection + */ + private void doCreateDatabase(List params) throws IOException { + if (params.isEmpty()) { + throw new IllegalArgumentException("Missing database template"); + } + else if (params.size() > 1) { + throw new IllegalArgumentException("Unexpected parameter: " + params.get(1)); + } + + String configTemplate = params.get(0); + boolean noTrackCallGraph = booleanOptions.contains(NO_CALLGRAPH_OPTION); + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.createDatabase(configTemplate, nameOption, ownerOption, descOption, + !noTrackCallGraph); + } + } + + private void doGenerateSigs(List params, TaskMonitor monitor) + throws Exception, CancelledException { + // concurrent bsim= and config= option use already checked + if (params.size() > 1) { + throw new IllegalArgumentException("Invalid generatesigs parameter use!"); + } + boolean commitOption = booleanOptions.contains(COMMIT_OPTION); + boolean overwriteOption = booleanOptions.contains(OVERWRITE_OPTION); + + String xmlDirectory = null; + if (params.size() == 1) { + xmlDirectory = params.get(0); + if (configOption != null && commitOption) { + throw new IllegalArgumentException( + "Invalid option use with config= option: " + COMMIT_OPTION); + } + } + else { + if (overwriteOption) { + throw new IllegalArgumentException("Invalid option use: " + OVERWRITE_OPTION); + } + commitOption = true; // assume DB commit using temp XML directory + } + + try (BulkSignatures bsim = getBulkSignatures()) { + if (commitOption) { + // Generate and commit signatures to BSim database + bsim.signatureRepo(ghidraURL, xmlDirectory, overwriteOption, monitor); + } + else { + // Generate sig XML files only + bsim.generateSignaturesFromServer(ghidraURL, xmlDirectory, overwriteOption, + configOption, monitor); + } + } + } + + private void doGenerateUpdates(List params, TaskMonitor monitor) + throws Exception, CancelledException { + // concurrent bsim= and config= option use already checked + if (params.size() > 1) { + throw new IllegalArgumentException("Invalid generateupdates parameter use!"); + } + boolean commitOption = booleanOptions.contains(COMMIT_OPTION); + boolean overwriteOption = booleanOptions.contains(OVERWRITE_OPTION); + + String xmlDirectory = null; + if (params.size() == 1) { + xmlDirectory = params.get(0); + if (configOption != null && commitOption) { + throw new IllegalArgumentException( + "Invalid option use with config= option: " + COMMIT_OPTION); + } + + } + else { + if (overwriteOption) { + throw new IllegalArgumentException("Invalid option use: " + OVERWRITE_OPTION); + } + commitOption = true; // assume DB commit using temp XML directory + } + + try (BulkSignatures bsim = getBulkSignatures()) { + if (commitOption) { + // Generate and commit updates to BSim database + bsim.updateRepoSignatures(ghidraURL, xmlDirectory, overwriteOption, monitor); + } + else { + // Generate update XML files only + bsim.generateUpdatesFromServer(ghidraURL, xmlDirectory, overwriteOption, + configOption, monitor); + } + } + } + + private File checkDirectory(String dirPath) throws IOException { + File dir = new File(dirPath); + if (!dir.exists()) { + throw new IOException("Commit directory does not exist: " + dirPath); + } + if (!dir.isDirectory()) { + throw new IOException(dirPath + ": is not a directory"); + } + return dir.getCanonicalFile(); + } + + private void doCommitSigs(List params, TaskMonitor monitor) + throws IOException, SAXException, LSHException, CancelledException { + if (params.size() < 1) { + throw new IllegalArgumentException("Missing directory containing signature files"); + } + if (!overrideOption && ghidraURLOption != null) { + throw new IllegalArgumentException( + "The \"ghidra=\" option use requires \"override\" option"); + } + String xmlDirectory = params.get(0); + + File dir = checkDirectory(xmlDirectory); + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.sendXmlToQueryServer(dir, overrideOption ? ghidraURL : null, filterOption, + monitor); + } + } + + private void doCommitUpdates(List params) + throws IOException, SAXException, LSHException { + if (params.size() < 1) { + throw new IllegalArgumentException("Missing directory containing update files"); + } + String xmlDirectory = params.get(0); + + File dir = checkDirectory(xmlDirectory); + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.sendUpdateToServer(dir); + } + } + + private boolean isAllNull(String... strings) { + for (String s : strings) { + if (s != null) { + return false; + } + } + return true; + } + + private void fillinSingleExeSpecifier(ExeSpecifier spec) throws IllegalArgumentException { + if (md5Option != null) { + if (!isAllNull(nameOption, archOption, compOption)) { + throw new IllegalArgumentException( + "The name=, arch= and compiler= options are not valid when md5= option is specified."); + } + spec.exemd5 = md5Option; + } + else if (nameOption != null) { + spec.exename = nameOption; + spec.arch = archOption; + spec.execompname = compOption; + } + else { + throw new IllegalArgumentException("Must specify either \"md5=\" or \"name=\" option"); + } + } + + private void doListFunctions(List params) throws IOException, LSHException { + + QueryName query = new QueryName(); + fillinSingleExeSpecifier(query.spec); + if (maxFunc != null) { + query.maxfunc = maxFunc; + } + if (booleanOptions.contains(PRINT_SELF_SIGNIFICANCE_OPTION)) { + query.printselfsig = true; + } + if (booleanOptions.contains(CALL_GRAPH_OPTION)) { + query.fillinCallgraph = true; + } + if (booleanOptions.contains(PRINT_JUST_EXE_OPTION)) { + query.printjustexe = true; + } + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.printFunctions(query, System.out); + } + } + + /** + * Deletes a specified executable from the database. + * + * @param params the command-line parameters + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doDeleteExecutable(List params) throws IOException, LSHException { + + ExeSpecifier spec = new ExeSpecifier(); + fillinSingleExeSpecifier(spec); + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.deleteExecutables(spec); + } + } + + /** + * Drops the current BSim database index. + * + * This variant of the drop index method should be called by + * clients using the command-line + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doDropIndex(List params) throws IOException, LSHException { + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.dropIndex(); + } + } + + /** + * Rebuilds the current BSim database index. + * + * This variant of the rebuild index method should be called by + * clients using the command-line + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doRebuildIndex(List params) throws IOException, LSHException { + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.rebuildIndex(); + } + } + + /** + * Performs a prewarm command on the BSim database. + *

+ * This is intended for use by command-line clients. + * + * @param params the command-line params (empty for this command) + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doPrewarm(List params) throws IOException, LSHException { + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.prewarm(); + } + } + + /** + * Display list of all executable records meeting a set of search criteria. + * Results are written to the output stream + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doListExes(List params) throws IOException, LSHException { + + int limit = DEFAULT_LIST_EXE_LIMIT; + if (limitOption != null) { + limit = limitOption; + } + boolean includeLibs = booleanOptions.contains(INCLUDE_LIBS_OPTION); + + try (BulkSignatures bsim = getBulkSignatures()) { + List exeList = bsim.getExes(limit, md5Option, nameOption, archOption, + compOption, sortColumn, includeLibs); + for (ExecutableRecord exeRec : exeList) { + Msg.info(this, exeRec.printRaw()); + } + } + } + + /** + * Print the number of records in the database that match the filter criteria. + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doGetCount(List params) throws IOException, LSHException { + boolean includeFakes = booleanOptions.contains(INCLUDE_LIBS_OPTION); + try (BulkSignatures bsim = getBulkSignatures()) { + int count = bsim.getCount(md5Option, nameOption, archOption, compOption, includeFakes); + System.out.println("Matching executable count: " + count); + } + } + + /** + * Updates the BSim database metadata with the given values. + * + * This variant of the update metadata method is intended for command-line + * users. + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doInstallMetadata(List params) throws IOException, LSHException { + if (isAllNull(nameOption, ownerOption, descOption)) { + throw new IllegalArgumentException( + "Missing one or more metadata options: [name=..] [owner=..] [description=..]"); + } + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.installMetadata(nameOption, ownerOption, descOption); + } + } + + /** + * Inserts a new category name into the BSim database. + * + * This variant of the install category method is intended for command-line + * users. + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doInstallCategory(List params) throws IOException, LSHException { + if (params.size() < 1) { + throw new IllegalArgumentException("Missing name of new category"); + } + boolean dateOption = booleanOptions.contains(CATEGORY_DATE_OPTION); + + String categoryName = params.get(0); + + if (params.size() > 1) { + throw new IllegalArgumentException("Unexpected parameter: " + params.get(1)); + } + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.installCategory(categoryName, dateOption); + } + } + + /** + * Inserts a new function tag into the BSim database. + * + * This variant of the tag install method is intended for command-line + * users. + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doInstallTags(List params) throws IOException, LSHException { + if (params.size() < 1) { + throw new IllegalArgumentException("Missing name of new function tag"); + } + if (params.size() > 1) { + throw new IllegalArgumentException("Unknown option: " + params.get(1)); + } + + String functionTag = params.get(0); + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.installTags(functionTag); + } + } + + /** + * Exports exe signature to local folder in XML format. + * + * @param params the command-line params + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + private void doDumpSigs(List params) throws IOException, LSHException { + if (params.size() < 1) { + throw new IllegalArgumentException("Must specify an output directory"); + } + File resultFolder = new File(params.get(0)); + + QueryName query = new QueryName(); + fillinSingleExeSpecifier(query.spec); + query.maxfunc = 0; // all functions + + try (BulkSignatures bsim = getBulkSignatures()) { + bsim.doDumpSigs(resultFolder, query); + } + } + + private static void printUsage() { + //@formatter:off + System.err.println( + "USAGE: bsim [command] required-args... [OPTIONS...]\n" + + " createdatabase [name=\"\"] [owner=\"\"] [description=\"\"] [--nocallgraph]\n" + + " setmetadata [name=\"\"] [owner=\"\"] [description=\"\"]\n" + + " addexecategory [--date]\n" + + " addfunctiontag \n" + + " dropindex \n" + + " rebuildindex \n" + + " prewarm \n" + + " generatesigs config= [--overwrite]\n" + + " generatesigs bsim= [--commit] [--overwrite]\n" + + " generatesigs bsim=\n" + + " commitsigs [md5=] [override=]\n" + + " generateupdates config= [--overwrite]\n" + + " generateupdates bsim= [--commit] [--overwrite]\n" + + " generateupdates bsim=\n" + + " commitupdates \n" + + " listexes [md5=] [name=] [arch=] [compiler=] [sortcol=] [limit=] [--includelibs]\n" + + " getexecount [md5=] [name=] [arch=] [compiler=] [--includelibs]\n" + + " delete [md5=] [name= [arch=] [compiler=]]\n" + + " listfuncs [md5=] [name= [arch=] [compiler=]] [--printselfsig] [--callgraph] [--printjustexe] [maxfunc=]\n" + + " dumpsigs [md5=] [name= [arch=] [compiler=]]\n" + + "\n" + + "Global options:\n" + + " user=\n" + + " cert=\n" + + "\n" + + "Enumerated Options:\n" + + " - large_32 | medium_32 | medium_64 | medium_cpool | medium_nosize \n" + + "\n" + + "BSim URL Forms (bsimURL):\n" + + " postgresql://[:]/\n" + + " elastic://[:]/\n" + + " https://[:]/\n" + + " file:/[/]\n" + + "\n" + + "Ghidra URL Forms (ghidraURL):\n" + + " ghidra://[:]/[/]\n" + + " ghidra:/[/][?/]\n" + + "\n"); + //@formatter:on + } + + @Override + public void launch(GhidraApplicationLayout ghidraLayout, String[] params) { + if (params.length == 0) { + printUsage(); + return; + } + + layout = ghidraLayout; + + try { + run(params); + } + catch (MalformedURLException e) { + Msg.error(this, "Invalid URL specified: " + e.getMessage()); + System.exit(22); // EINVAL + } + catch (IllegalArgumentException e) { + Msg.error(this, e.getMessage()); + System.out.println("Execute \"bsim\" without arguments to display usage details"); + System.exit(22); // EINVAL + } + catch (Exception e) { + Msg.error(this, e.getMessage()); + System.exit(1); // Misc Error + } + } + + private void initializeApplication(String command) throws IOException { + int initType = COMMANDS_WITH_REPO_ACCESS.contains(command) ? 2 : 1; + if (layout != null) { + initializeApplication(layout, initType, connectingUserName, certOption); + } + } + + /** + * From a cold start, initialize the Ghidra application to different stages, based on future requirements + * @param layout application layout + * @param type is an integer indicating how much to initialize + * 0 - limited initialization, enough simple execution and logging + * 1 - full initialization of ghidra for module path info and initialization + * 2 - same as #1 with class search for extensions + * @param connectingUserName default user name for server connections + * @param certPath PKI certificate path + * @throws IOException if there is a problem initializing the headless authenticator + */ + public static void initializeApplication(ApplicationLayout layout, int type, + String connectingUserName, String certPath) throws IOException { + if (Application.isInitialized()) { + return; + } + + /** + * Ensure that we are running in "headless mode" + */ + System.setProperty(SystemUtilities.HEADLESS_PROPERTY, Boolean.TRUE.toString()); + + try { + URL configFileUrl = + BSimLaunchable.class.getClassLoader().getResource(BSIM_LOGGING_CONFIGURATION_FILE); + System.setProperty(LoggingInitialization.LOG4J2_CONFIGURATION_PROPERTY, + configFileUrl.toURI().toString()); + } + catch (URISyntaxException e) { + System.err.println("ERROR: " + e.getMessage()); + } + + // Allows handling of old content which did not have a content type property + DomainObjectAdapter.setDefaultContentClass(ProgramDB.class); + + ApplicationConfiguration config; + switch (type) { + case 2: + // application support with class searching and extensions (e.g., ContentHandler) + config = new HeadlessGhidraApplicationConfiguration(); + break; + + case 1: + // application support without class searching and extensions + config = new HeadlessBSimApplicationConfiguration(); + break; + + default: + // Setup application with minimal application support + config = new ApplicationConfiguration(); + } + + Application.initializeApplication(layout, config); + + SSLContextInitializer.initialize(); + ghidra.framework.protocol.ghidra.Handler.registerHandler(); + ghidra.features.bsim.query.postgresql.Handler.registerHandler(); + + HeadlessClientAuthenticator.installHeadlessClientAuthenticator(connectingUserName, certPath, + true); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/BulkSignatures.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/BulkSignatures.java new file mode 100755 index 0000000000..17a91b39f1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/BulkSignatures.java @@ -0,0 +1,1120 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.ingest; + +import java.io.*; +import java.net.MalformedURLException; +import java.net.URL; +import java.util.*; + +import javax.help.UnsupportedOperationException; + +import org.apache.commons.lang3.StringUtils; +import org.xml.sax.ErrorHandler; +import org.xml.sax.SAXException; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.app.decompiler.DecompileException; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.FunctionDatabase.Error; +import ghidra.features.bsim.query.FunctionDatabase.ErrorCategory; +import ghidra.features.bsim.query.FunctionDatabase.Status; +import ghidra.features.bsim.query.client.Configuration; +import ghidra.features.bsim.query.client.tables.ExeTable.ExeTableOrderColumn; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.*; +import ghidra.features.bsim.query.protocol.ResponseDelete.DeleteResult; +import ghidra.framework.client.ClientUtil; +import ghidra.framework.protocol.ghidra.GhidraURL; +import ghidra.program.model.listing.*; +import ghidra.util.Msg; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.*; + +public class BulkSignatures implements AutoCloseable { + + // FIXME: May need to use Msg.showError for popup messages in GUI workbench case + + private final BSimServerInfo bsimServerInfo; // may be null + private final String connectingUserName; + + private FunctionDatabase querydb; + + /** + * Constructor + * @param bsimServerInfo the BSim database server info. May be {@code null} if use limited to + * signature and update generation only (based upon configuration template). + * @param connectingUserName user name to use for BSim server authentication. May be null if + * not required or default should be used (see {@link ClientUtil#getUserName()}). + * @throws MalformedURLException if the given URL string cannot be parsed + */ + public BulkSignatures(BSimServerInfo bsimServerInfo, String connectingUserName) + throws MalformedURLException { + this.bsimServerInfo = bsimServerInfo; + this.connectingUserName = + connectingUserName != null ? connectingUserName : ClientUtil.getUserName(); + } + + private void checkBSimServerOperation() { + if (bsimServerInfo == null) { + throw new UnsupportedOperationException("BSim server has not been specified"); + } + } + + /** + * + * + * @param async true if database commits should be synchronous + * @return the {@link DatabaseInformation} object returned from a successful connect + * @throws IOException if there's a problem creating the connection + */ + private DatabaseInformation establishQueryServerConnection(boolean async) throws IOException { + + if (querydb != null) { + return querydb.getInfo(); + } + + checkBSimServerOperation(); + + querydb = BSimClientFactory.buildClient(bsimServerInfo, async); + if (querydb.getStatus() == Status.Unconnected) { // may have previously connected + querydb.setUserName(connectingUserName); + } + + if (!querydb.initialize()) { + throw new IOException(querydb.getLastError().message); + } + + DatabaseInformation info = querydb.getInfo(); + if (info == null) { + Error lastError = querydb.getLastError(); + if (lastError != null && lastError.category == ErrorCategory.Nodatabase) { + throw new IOException(lastError.message); + } + throw new IOException("Unknown error connection to: " + bsimServerInfo.toString()); + } + + Msg.debug(this, "Connected to " + info.databasename); + return info; + } + + /** + * This will be automatically invoked when BulkSignatures is out of scope, if using + * try-with-resources to create it. When this happens we need to clean up the + * connection. + */ + @Override + public void close() { + closeConnection(); + } + + /** + * Closes the current database connection. + */ + private void closeConnection() { + if (querydb != null) { + querydb.close(); + querydb = null; + } + } + + private List gatherXml(String prefix, File dir) throws IOException { + List res = new ArrayList(); + File[] listFiles = dir.listFiles(); + if (listFiles == null) { + throw new IOException("Bad xml directory"); + } + for (File file : listFiles) { + if (file.getName().startsWith(prefix)) { + res.add(file); + } + } + return res; + } + + private void loadSignatureXml(File file, DescriptionManager manage) + throws SAXException, IOException, LSHException { + ErrorHandler handler = SpecXmlUtils.getXmlHandler(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(file, handler, false); + manage.restoreXml(parser, querydb.getLSHVectorFactory()); + } + + protected void sendXmlToQueryServer(File dir, URL ghidraOverrideURL, String filter, + TaskMonitor monitor) + throws IOException, SAXException, LSHException, CancelledException { + establishQueryServerConnection(true); + if (filter == null) { + filter = "sigs_"; + } + else { + filter = "sigs_" + filter; + } + List files = gatherXml(filter, dir); + + if (files.isEmpty()) { + throw new IOException("No signature files found in " + dir.getAbsolutePath()); + } + + monitor.setMessage("Writing signatures"); + monitor.setMaximum(files.size()); + + for (File file : files) { + monitor.checkCancelled(); + monitor.incrementProgress(1); + Msg.info(this, "Writing signatures for " + file.getName()); + InsertRequest insertreq = new InsertRequest(); + if (ghidraOverrideURL != null) { + insertreq.repo_override = + GhidraURL.getProjectURL(ghidraOverrideURL).toExternalForm(); + insertreq.path_override = GhidraURL.getProjectPathname(ghidraOverrideURL); + } + loadSignatureXml(file, insertreq.manage); + if (insertreq.execute(querydb) == null) { + Error lastError = querydb.getLastError(); + if ((lastError.category == ErrorCategory.Format) || + (lastError.category == ErrorCategory.Nonfatal)) { + Msg.warn(this, file.getName() + ": " + lastError.message); + } + else { + throw new IOException(file.getName() + ": " + lastError.message); + } + } + } + } + + protected void sendUpdateToServer(File dir) throws IOException, SAXException, LSHException { + establishQueryServerConnection(true); + List files = gatherXml("update_", dir); + + if (files.isEmpty()) { + throw new IOException("No update files found in " + dir.getAbsolutePath()); + } + + for (File file : files) { + Msg.info(this, "Updating metadata for " + file.getName()); + QueryUpdate update = new QueryUpdate(); + loadSignatureXml(file, update.manage); + ResponseUpdate respup = update.execute(querydb); + if (respup == null) { + Error lastError = querydb.getLastError(); + if ((lastError.category == ErrorCategory.Format) || + (lastError.category == ErrorCategory.Nonfatal)) { + Msg.warn(this, file.getName() + ": " + lastError.message); + } + else { + throw new IOException(file.getName() + ": " + lastError.message); + } + } + else { + if (!respup.badexe.isEmpty()) { + for (ExecutableRecord erec : respup.badexe) { + Msg.error(this, "Unable to find executable: " + erec.getNameExec()); + } + } + if (!respup.badfunc.isEmpty()) { + int max = respup.badfunc.size(); + if (max > 3) { + Msg.error(this, "Could not find " + + Integer.toString(respup.badfunc.size()) + " functions"); + max = 3; + } + for (int j = 0; j < max; ++j) { + FunctionDescription func = respup.badfunc.get(j); + Msg.error(this, "Could not update function " + func.getFunctionName()); + } + } + if (respup.exeupdate > 0) { + Msg.info(this, + "Updated " + Integer.toString(respup.exeupdate) + " executables"); + } + if (respup.funcupdate > 0) { + Msg.info(this, "Updated " + Integer.toString(respup.funcupdate) + " functions"); + } + if (respup.exeupdate == 0 && respup.funcupdate == 0) { + Msg.info(this, "No changes"); + } + } + } + } + + private DatabaseInformation createQueryDatabase(String template, String name, String owner, + String description, boolean track) throws IOException { + CreateDatabase command = new CreateDatabase(); + command.info = new DatabaseInformation(); + // Put in fields provided on the command line + // If they are null, the template will fill them in + command.info.databasename = name; + command.info.owner = owner; + command.info.description = description; + command.config_template = template; + command.info.trackcallgraph = track; + ResponseInfo response = command.execute(querydb); + if (response == null) { + throw new IOException(querydb.getLastError().message); + } + return response.info; + } + + private void formatCategories(List execats, StringBuilder buf) { + if (execats == null) { + return; + } + buf.append(" Categories:\n"); + for (String execat : execats) { + buf.append(" "); + buf.append(execat); + buf.append("\n"); + } + } + + private void formatFunctionTags(List tags, StringBuilder buf) { + if (tags == null) { + return; + } + buf.append(" Function Tags:\n"); + for (String tag : tags) { + buf.append(" "); + buf.append(tag); + buf.append("\n"); + } + } + + private String formatDatabaseDetails(DatabaseInformation info) { + StringBuilder buf = new StringBuilder(); + buf.append("Using configuration for:\n"); + buf.append(" Database: "); + buf.append(info.databasename); + buf.append("\n"); + buf.append(" Owner: "); + buf.append(info.owner); + buf.append("\n"); + formatCategories(info.execats, buf); + formatFunctionTags(info.functionTags, buf); + if (info.dateColumnName != null) { + buf.append(" Date column: "); + buf.append(info.dateColumnName); + buf.append("\n"); + } + return buf.toString(); + } + + protected File generateSignaturesFromServer(URL ghidraURL, String xmlDirectory, + boolean overwrite, String configtemplate, TaskMonitor monitor) + throws Exception, CancelledException { + + File dir = establishTemporaryDirectory(xmlDirectory); + + DatabaseInformation info; + LSHVectorFactory vectorFactory = null; + if (configtemplate == null) { + info = establishQueryServerConnection(false); + Msg.debug(this, "Attempting to pull configuration from: " + bsimServerInfo.toString()); + vectorFactory = querydb.getLSHVectorFactory(); + } + else { + // User gave an overriding configuration name on the command line + Configuration config = FunctionDatabase.loadConfigurationTemplate(configtemplate); + info = config.info; + vectorFactory = FunctionDatabase.generateLSHVectorFactory(); + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + } + + // TODO: Should this output differ for command-line vs workbench? debug only? + Msg.info(this, formatDatabaseDetails(info)); + + String repositoryURLString = GhidraURL.getProjectURL(ghidraURL).toExternalForm(); + SignatureRepository sigrepo = + new SignatureRepository(dir, repositoryURLString, overwrite, info, vectorFactory); + + sigrepo.process(ghidraURL, monitor); + + return dir; + } + + protected File generateUpdatesFromServer(URL ghidraURL, String xmlDirectory, boolean overwrite, + String configtemplate, TaskMonitor monitor) throws Exception, CancelledException { + + File dir = establishTemporaryDirectory(xmlDirectory); + + DatabaseInformation info; + LSHVectorFactory vectorFactory = null; + if (configtemplate == null) { + info = establishQueryServerConnection(false); + vectorFactory = querydb.getLSHVectorFactory(); + } + else { + // User gave an overriding configuration name on the command line + Configuration config = FunctionDatabase.loadConfigurationTemplate(configtemplate); + info = config.info; + vectorFactory = FunctionDatabase.generateLSHVectorFactory(); + vectorFactory.set(config.weightfactory, config.idflookup, config.info.settings); + } + + // TODO: Should this output differ for command-line vs workbench? debug only? + Msg.info(this, formatDatabaseDetails(info)); + + String repositoryURLString = GhidraURL.getProjectURL(ghidraURL).toExternalForm(); + UpdateRepository updaterepo = + new UpdateRepository(dir, repositoryURLString, overwrite, info, vectorFactory); + + updaterepo.process(ghidraURL, monitor); + + return dir; + } + + /** + * Creates a new BSim database with a given set of properties. + * + * @param configTemplate the type of database to create + * @param name the name of the database + * @param owner the owner of the database + * @param description the database description + * @param trackCall if true, the database should track callgraph information + * @throws IOException if there's an error building the {@link BSimClientFactory} + */ + public void createDatabase(String configTemplate, String name, String owner, String description, + boolean trackCall) throws IOException { + + closeConnection(); + + checkBSimServerOperation(); + + querydb = BSimClientFactory.buildClient(bsimServerInfo, true); + if (querydb.getStatus() == Status.Unconnected) { // may have previously connected + querydb.setUserName(connectingUserName); + } + + // TODO: Should this output differ for command-line vs workbench? debug only? + try { + DatabaseInformation info = + createQueryDatabase(configTemplate, name, owner, description, trackCall); + + StringBuilder buf = new StringBuilder(); + buf.append("Created database: "); + buf.append(info.databasename); + buf.append("\n"); + buf.append(" owner = "); + buf.append(info.owner); + buf.append("\n"); + buf.append(" description = "); + buf.append(info.description); + buf.append("\n"); + buf.append(" template = "); + buf.append(configTemplate); + buf.append("\n"); + + // TODO: Should this output differ for command-line vs workbench? debug only? + Msg.info(this, buf.toString()); + } + catch (IOException ex) { + Msg.error(this, "Unable to create database: " + ex.getMessage()); + return; + } + + // Always attempt to initialize after creation. + boolean success = querydb.initialize(); + if (!success) { + Msg.error(this, "Database initialization error: " + querydb.getLastError().message); + } + } + + /** + * Adds function signatures from the specified project to the BSim database + * @param ghidraURL ghidra repository from which to pull files for signature generation + * @param sigsLocation the location where signature files will be stored + * @param overwrite if true, overwrites any existing signatures + * @param monitor the task monitor + * @throws Exception if there's an error during the operation + * @throws CancelledException if processing is cancelled + */ + public void signatureRepo(URL ghidraURL, String sigsLocation, boolean overwrite, + TaskMonitor monitor) throws Exception, CancelledException { + + String xmlDirectory = null; + boolean usestmpdir = false; + + if (!StringUtils.isBlank(sigsLocation)) { + xmlDirectory = sigsLocation; + } + else { + usestmpdir = true; + } + + File dir = generateSignaturesFromServer(ghidraURL, xmlDirectory, overwrite, null, monitor); + sendXmlToQueryServer(dir, null, null, monitor); + if (usestmpdir) { + deleteTemporaryDirectory(dir); + } + } + + /** + * Updates function signatures from the specified project to the BSim database + * @param ghidraURL ghidra repository from which to pull files for signature generation + * @param sigsLocation the location where update XML files are + * @param overwrite if true, overwrites any existing signatures + * @param monitor the task monitor + * @throws Exception if there's an error during the operation + * @throws CancelledException if processing is cancelled + */ + public void updateRepoSignatures(URL ghidraURL, String sigsLocation, boolean overwrite, + TaskMonitor monitor) throws Exception, CancelledException { + + String xmlDirectory = null; + boolean usestmpdir = false; + + if (!StringUtils.isAnyBlank(sigsLocation)) { + xmlDirectory = sigsLocation; + } + else { + usestmpdir = true; + } + + File dir = generateUpdatesFromServer(ghidraURL, xmlDirectory, overwrite, null, monitor); + sendUpdateToServer(dir); + if (usestmpdir) { + deleteTemporaryDirectory(dir); + } + } + + /** + * Deletes a specified executable from the database. + * + * @param md5 the MD5 of the executable to delete + * @param name the name of the executable to delete + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + public void deleteExecutable(String md5, String name) throws IOException, LSHException { + + if (StringUtils.isAnyBlank(md5) && StringUtils.isAnyBlank(name)) { + throw new IOException("Must specify \"md5=\" or \"name=\" option"); + } + + ExeSpecifier spec = new ExeSpecifier(); + spec.exemd5 = md5; + spec.exename = name; + spec.arch = null; + spec.execompname = null; + + deleteExecutables(spec); + } + + /** + * Deletes a specified executable from the database. + * + * @param spec the spec that indicates what executable to delete + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + protected void deleteExecutables(ExeSpecifier spec) throws IOException, LSHException { + + QueryDelete query = new QueryDelete(); + query.addSpecifier(spec); + establishQueryServerConnection(true); + ResponseDelete respdel = query.execute(querydb); + if (respdel == null) { + Error lastError = querydb.getLastError(); + throw new LSHException("Could not perform delete: " + lastError.message); + } + + // TODO: Should this output differ for command-line vs workbench? debug only? + for (DeleteResult delres : respdel.reslist) { + Msg.info(this, "Successfully deleted " + delres.name + "(" + + Integer.toString(delres.funccount) + " functions)" + delres.md5); + } + for (ExeSpecifier missedSpec : respdel.missedlist) { + Msg.error(this, "Unable to uniquely identify: " + missedSpec.getExeNameWithMD5()); + } + } + + /** + * Drops the current BSim database index which can allow for faster signature ingest after + * which a {@link #rebuildIndex()} may be performed. Dropping the index may also be done to + * obtain more accurate results albeit at the cost of performance. + * + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + public void dropIndex() throws IOException, LSHException { + DatabaseInformation info = establishQueryServerConnection(false); + AdjustVectorIndex query = new AdjustVectorIndex(); + query.doRebuild = false; + ResponseAdjustIndex response = query.execute(querydb); + if (response == null) { + Error lastError = querydb.getLastError(); + throw new LSHException("Could not drop index: " + lastError.message); + } + String dbDetail = "for database " + info.databasename + " (" + bsimServerInfo + ")"; + if (!response.success) { + String msg = "Could not drop the index " + dbDetail; + if (!response.operationSupported) { + msg += ": operation not supported"; + } + Msg.error(this, msg); + } + else { + Msg.info(this, "Successfully dropped index " + dbDetail); + } + } + + /** + * Rebuilds the current BSim database index. + * + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + public void rebuildIndex() throws IOException, LSHException { + DatabaseInformation info = establishQueryServerConnection(false); + AdjustVectorIndex query = new AdjustVectorIndex(); + query.doRebuild = true; + System.out.println("Starting rebuild ..."); + ResponseAdjustIndex response = query.execute(querydb); + if (response == null) { + Error lastError = querydb.getLastError(); + throw new LSHException("Could not rebuild index: " + lastError.message); + } + String dbDetail = "for database " + info.databasename + " (" + bsimServerInfo + ")"; + if (!response.success) { + String msg = "Could not rebuild index " + dbDetail; + if (!response.operationSupported) { + msg += ": operation not supported"; + } + Msg.error(this, msg); + } + else { + Msg.info(this, "Successfully rebuilt index " + dbDetail); + } + } + + /** + * Performs a prewarm command on the BSim database. + * + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + public void prewarm() throws IOException, LSHException { + DatabaseInformation info = establishQueryServerConnection(false); + PrewarmRequest request = new PrewarmRequest(); + ResponsePrewarm response = request.execute(querydb); + if (response == null) { + Error lastError = querydb.getLastError(); + throw new LSHException("Prewarm failed: " + lastError.message); + } + String dbDetail = "for database " + info.databasename + " (" + bsimServerInfo + ")"; + if (!response.operationSupported) { + Msg.error(this, "Prewarm operation not supported " + dbDetail); + } + else { + Msg.info(this, "Successfully prewarmed " + Integer.toString(response.blockCount) + + " blocks of main index " + dbDetail); + } + } + + /** + * Returns a list of all executable records meeting a set of search criteria. + * + * @param limit the maximum number of results to return + * @param md5Filter MD5 filter + * @param exeNameFilter executable name filter + * @param archFilter architecture filter + * @param compilerFilter compiler name filter + * @param sortCol the main sort column (either name or md5) + * @param incFakes if true, include executables with an MD5 that we generate + * @return the list of executables matching the filters + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + protected List getExes(int limit, String md5Filter, String exeNameFilter, + String archFilter, String compilerFilter, String sortCol, boolean incFakes) + throws IOException, LSHException { + + establishQueryServerConnection(false); + ExeTableOrderColumn sortEnum; + if (sortCol != null) { + sortEnum = ExeTableOrderColumn.valueOf(sortCol); + } + else { + sortEnum = ExeTableOrderColumn.MD5; + } + + QueryExeInfo exeQuery = new QueryExeInfo(limit, md5Filter, exeNameFilter, archFilter, + compilerFilter, sortEnum, incFakes); + + ResponseExe response = exeQuery.execute(querydb); + if (response == null) { + Error lastError = querydb.getLastError(); + throw new LSHException("Could not perform getexeinfo: " + lastError.message); + } + + return response.records; + } + + /** + * Retrieves the number of records in the database that match the filter criteria. + * + * @param md5Filter the MD5 value must contain this + * @param exeNameFilter the executable name must contain this + * @param archFilter the architecture type must match this + * @param compilerFilter the compiler type must match this + * @param incFakes if true, include executables with an MD5 that we created + * @return the number of executables matching the filter criteria + * @throws IOException if there's a problem establishing the database connection + */ + public int getCount(String md5Filter, String exeNameFilter, String archFilter, + String compilerFilter, boolean incFakes) throws IOException { + + establishQueryServerConnection(false); + + QueryExeCount exeQuery = + new QueryExeCount(md5Filter, exeNameFilter, archFilter, compilerFilter, incFakes); + ResponseExe response = exeQuery.execute(querydb); + if (response == null) { + return 0; + } + + return response.recordCount; + } + + /** + * Remove one layer of quoting + * @param val is the string which might be quoted + * @return the string with any outer quote characters stripped + */ + protected static String dequoteString(String val) { + if (val.length() < 3) { + return val; + } + if (val.charAt(0) != '\"') { + return val; + } + if (val.charAt(val.length() - 1) != '\"') { + return val; + } + val = val.substring(1, val.length() - 1); + return val; + } + + /** + * Performs the work of updating the metadata. This will build the query + * object, establish the database connection, and perform the query. + * + * @param name the database name + * @param owner the database owner + * @param description the database description + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + protected void installMetadata(String name, String owner, String description) + throws IOException, LSHException { + + DatabaseInformation info = establishQueryServerConnection(false); + + InstallMetadataRequest req = new InstallMetadataRequest(); + req.dbname = name; + req.owner = owner; + req.description = description; + ResponseInfo resp = req.execute(querydb); + if (resp == null) { + Error lastError = querydb.getLastError(); + throw new LSHException("Could not change metadata: " + lastError.message); + } + info = resp.info; + Msg.info(this, "Updated BSim metadata: "); + Msg.info(this, " Database: " + info.databasename); + Msg.info(this, " Owner: " + info.owner); + Msg.info(this, " Description: " + info.description); + } + + /** + * Performs the work of installing a new category name. This will build the query + * object, establish the database connection, and perform the query. + * + * @param categoryName the category name to insert + * @param isDate true if this is a date category + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + public void installCategory(String categoryName, boolean isDate) + throws LSHException, IOException { + DatabaseInformation info = establishQueryServerConnection(false); + + InstallCategoryRequest req = new InstallCategoryRequest(); + req.type_name = dequoteString(categoryName); + req.isdatecolumn = isDate; + + ResponseInfo resp = req.execute(querydb); + if (resp == null) { + Error lastError = querydb.getLastError(); + throw new LSHException("Could not install new category: " + lastError.message); + } + info = resp.info; + + StringBuilder buf = new StringBuilder(); + buf.append("BSim Database "); + buf.append(info.databasename); + buf.append(" now contains:\n"); + formatCategories(info.execats, buf); + if (info.dateColumnName != null) { + buf.append(" Date column: "); + buf.append(info.dateColumnName); + buf.append("\n"); + } + + // TODO: Should this output differ for command-line vs workbench? debug only? + Msg.info(this, buf.toString()); + } + + /** + * Performs the work of inserting a new function tag name into the database. This + * will build the query object, establish the database connection, and perform the query. + * + * @param tagName the tag name to insert + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + public void installTags(String tagName) throws IOException, LSHException { + DatabaseInformation info = establishQueryServerConnection(false); + InstallTagRequest req = new InstallTagRequest(); + req.tag_name = dequoteString(tagName); + ResponseInfo resp = req.execute(querydb); + if (resp == null) { + Error lastError = querydb.getLastError(); + throw new LSHException(lastError.message); + } + info = resp.info; + + StringBuilder buf = new StringBuilder(); + buf.append("BSim Database "); + buf.append(info.databasename); + buf.append(" now contains:\n"); + formatFunctionTags(info.functionTags, buf); + + // TODO: Should this output differ for command-line vs workbench? debug only? + Msg.info(this, buf.toString()); + } + + protected static int readQueryPairs(XmlPullParser parser, int count, List pairs) { + for (int i = 0; i < count; ++i) { + if (!parser.peek().isStart()) { + return i; + } + PairInput pairInput = new PairInput(); + pairInput.restoreXml(parser); + pairs.add(pairInput); + } + return count; + } + + /** + * Compares pairs of functions specified in an input (XML) file, and writes + * the results to an output file. + * + * @param inputFile input XML file + * @param outputFile output XML file + * @throws IOException if there is a problem establishing the server connection + * @throws SAXException if an XML parse error occurs + * @throws LSHException if there is a problem querying the database + */ + protected void queryPair(File inputFile, File outputFile) + throws IOException, SAXException, LSHException { + if (!inputFile.isFile()) { + throw new IOException(inputFile.getAbsolutePath() + " is not an XML file"); + } + if (outputFile.isFile()) { + Msg.info(this, "Overwriting file " + outputFile.getAbsolutePath()); + outputFile.delete(); + } + establishQueryServerConnection(true); + QueryPair query = new QueryPair(); + query.pairs = new ArrayList(); + ErrorHandler handler = SpecXmlUtils.getXmlHandler(); + XmlPullParser parser = XmlPullParserFactory.create(inputFile, handler, false); + parser.start("querypair"); + + try (FileWriter writer = new FileWriter(outputFile)) { + writer.append("\n"); + ResponsePair.Accumulator accumulator = new ResponsePair.Accumulator(); + ResponsePair finalResponse = new ResponsePair(); + int count = readQueryPairs(parser, 20, query.pairs); + while (count != 0) { + ResponsePair responsePair = query.execute(querydb); + if (responsePair == null) { + Error lastError = querydb.getLastError(); + throw new LSHException(lastError.message); + } + for (PairNote note : responsePair.notes) { + note.saveXml(writer); + } + finalResponse.scale = responsePair.scale; + accumulator.merge(responsePair); + query.pairs.clear(); + query.clearResponse(); + count = readQueryPairs(parser, 20, query.pairs); + } + parser.end(); + finalResponse.fillOutStatistics(accumulator); + finalResponse.saveXmlTail(writer); + writer.append("\n"); + } + } + + /** + * Execute the specified {@link QueryName} query and print the formatted results to the + * specified {@code outStream}. + * + * @param query function name query + * @param outStream stream to receive formatted output + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + protected void printFunctions(QueryName query, PrintStream outStream) + throws IOException, LSHException { + + establishQueryServerConnection(true); + ResponseName resp = query.execute(querydb); + if (resp == null) { + Error lastError = querydb.getLastError(); + throw new LSHException(lastError.message); + } + resp.printRaw(outStream, querydb.getLSHVectorFactory(), 0); + } + + /** + * Exports information about a binary to a local folder in XML format. + * + * @param resultFolder the folder where the results will be stored + * @param md5 the MD5 of the executables to export + * @param name the name of the executables to export + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + public void dumpSigs(File resultFolder, String md5, String name) + throws IOException, LSHException { + + if (StringUtils.isAnyBlank(md5) && StringUtils.isAnyBlank(name)) { + throw new IOException("Must specify \"md5=\" or \"name=\""); + } + + QueryName query = new QueryName(); + query.spec.exemd5 = md5; + query.spec.exename = name; + query.spec.arch = null; + query.spec.execompname = null; + + doDumpSigs(resultFolder, query); + } + + /** + * Exports information about a binary to a local folder in XML format. + * + * @param resultFolder the folder where the results will be stored + * @param query the query object containing the params of the query + * @throws IOException if there's an error establishing the database connection + * @throws LSHException if there's an error issuing the query + */ + protected void doDumpSigs(File resultFolder, QueryName query) throws IOException, LSHException { + if (!resultFolder.isDirectory()) { + throw new IOException(resultFolder.getAbsolutePath() + " is not a valid directory"); + } + + DatabaseInformation info = establishQueryServerConnection(true); + query.fillinCallgraph = info.trackcallgraph; + ResponseName responseName = query.execute(querydb); + if (responseName == null) { + Error lastError = querydb.getLastError(); + throw new LSHException(lastError.message); + } + if (!responseName.uniqueexecutable) { + throw new LSHException("Could not determine unique executable"); + } + ExecutableRecord exe; + if (!StringUtils.isAllBlank(query.spec.exemd5)) { + exe = responseName.manage.findExecutable(query.spec.exemd5); + } + else { + exe = responseName.manage.findExecutable(query.spec.exename, query.spec.arch, + query.spec.execompname); + } + String basename = "sigs_" + exe.getMd5(); + File sigFile = new File(resultFolder, basename); + + try (FileWriter writer = new FileWriter(sigFile)) { + responseName.manage.saveXml(writer); + } + } + + protected File establishTemporaryDirectory(String xmldir) throws IOException { + File dir; + if (xmldir == null) { + String tempDirString = System.getProperty("java.io.tmpdir"); + if (tempDirString == null) { + throw new IOException("Could not find temporary directory"); + } + dir = new File(tempDirString, "bulkinsert_xml"); + } + else { + dir = new File(xmldir); + } + if (dir.exists() == false) { + if (dir.mkdir() == false) { + throw new IOException("Unable to create temp directory: " + dir.getAbsolutePath()); + } + } + else if (dir.isDirectory() == false) { + throw new IOException(dir.getAbsolutePath() + ": is not a directory"); + } + dir = dir.getCanonicalFile(); + return dir; + } + + private void deleteTemporaryDirectory(File tempDir) throws IOException { + File[] listFiles = tempDir.listFiles(); + if (listFiles == null) { + throw new IOException( + "Could not list files in temp directory: " + tempDir.getAbsolutePath()); + } + for (File listFile : listFiles) { + if (!listFile.delete()) { + throw new IOException( + "Unable to delete temporary file: " + listFile.getAbsolutePath()); + } + } + if (!tempDir.delete()) { + throw new IOException("Unable to delete temp directory: " + tempDir.getAbsolutePath()); + } + } + + private class UpdateRepository extends IterateRepository { + private File outdirectory; + private String repo; + private boolean overwrite; + private DatabaseInformation info; + private LSHVectorFactory vectorFactory; + + public UpdateRepository(File outdir, String rp, boolean owrite, DatabaseInformation i, + LSHVectorFactory vFactory) { + outdirectory = outdir; + repo = rp; + overwrite = owrite; + info = i; + vectorFactory = vFactory; + } + + @Override + protected void process(Program program, TaskMonitor monitor) + throws IOException, LSHException { + // NOTE: task monitor not used by DescriptionManager + String md5string = program.getExecutableMD5(); + if ((md5string == null) || (md5string.length() < 10)) { + Msg.error(this, "Could not get MD5 on file: " + program.getDomainFile().getName()); + return; + } + String basename = "update_" + md5string; + File file = new File(outdirectory, basename); + if ((!overwrite) && file.exists()) { + Msg.warn(this, + "Update file already exists for: " + program.getDomainFile().getName()); + return; + } + GenSignatures gensig = new GenSignatures(true); + try { + gensig.setVectorFactory(vectorFactory); + gensig.addExecutableCategories(info.execats); + gensig.addFunctionTags(info.functionTags); + gensig.addDateColumnName(info.dateColumnName); + Msg.info(this, "Generating metadata for: " + program.getDomainFile().getName()); + String path = GenSignatures.getPathFromDomainFile(program); + gensig.openProgram(program, null, null, null, repo, path); + gensig.scanFunctionsMetadata(null, null); + DescriptionManager manager = gensig.getDescriptionManager(); + if (manager.numFunctions() == 0) { + Msg.warn(this, + program.getDomainFile().getName() + " contains no functions with bodies"); + } + try (FileWriter fwrite = new FileWriter(file)) { + manager.saveXml(fwrite); + } + } + catch (LSHException | IOException e) { + Msg.error(this, e.getMessage()); + throw e; + } + } + } + + private class SignatureRepository extends IterateRepository { + + private File outdirectory; + private String repo; // Repository URL to include with signature metadata + private boolean overwrite; // True if existing signature files should be overwritten + private DatabaseInformation info; // Database configuration (may affect signature generation) + private LSHVectorFactory vectorFactory; + + public SignatureRepository(File outdir, String rp, boolean owrite, DatabaseInformation i, + LSHVectorFactory vFactory) { + outdirectory = outdir; + repo = rp; + overwrite = owrite; + info = i; + vectorFactory = vFactory; + } + + @Override + protected void process(Program program, TaskMonitor monitor) + throws IOException, LSHException, DecompileException { + // NOTE: task monitor not used by DescriptionManager + String md5string = program.getExecutableMD5(); + if ((md5string == null) || (md5string.length() < 10)) { + Msg.error(this, "Could not get MD5 on file: " + program.getDomainFile().getName()); + return; + } + String basename = "sigs_" + md5string; + File file = new File(outdirectory, basename); + if ((!overwrite) && file.exists()) { + Msg.warn(this, + "Signature file already exists for: " + program.getDomainFile().getName()); + return; + } + GenSignatures gensig = new GenSignatures(true); + try { + gensig.setVectorFactory(vectorFactory); + gensig.addExecutableCategories(info.execats); + gensig.addFunctionTags(info.functionTags); + gensig.addDateColumnName(info.dateColumnName); + Msg.info(this, "Generating signatures for: " + program.getDomainFile().getName()); + String path = GenSignatures.getPathFromDomainFile(program); + gensig.openProgram(program, null, null, null, repo, path); + FunctionManager fman = program.getFunctionManager(); + Iterator iterator = fman.getFunctions(true); + gensig.scanFunctions(iterator, fman.getFunctionCount(), null); + DescriptionManager manager = gensig.getDescriptionManager(); + if (manager.numFunctions() == 0) { + Msg.warn(this, program.getDomainFile().getName() + + " contains no functions with signatures"); + } + FileWriter fwrite = new FileWriter(file); + manager.saveXml(fwrite); + fwrite.close(); + } + catch (DecompileException | LSHException | IOException e) { + Msg.error(this, e.getMessage()); + throw e; + } + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/HeadlessBSimApplicationConfiguration.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/HeadlessBSimApplicationConfiguration.java new file mode 100644 index 0000000000..3c14e7ccc4 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/HeadlessBSimApplicationConfiguration.java @@ -0,0 +1,63 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.ingest; + +import java.io.File; +import java.util.List; + +import generic.jar.ResourceFile; +import ghidra.framework.*; +import ghidra.net.ApplicationTrustManagerFactory; +import ghidra.util.classfinder.ClassSearcher; + +public class HeadlessBSimApplicationConfiguration extends ApplicationConfiguration { + + @Override + protected void initializeApplication() { + super.initializeApplication(); + + // Locate certs if found (must be done before module initialization) + locateCACertsFile(); + + monitor.setMessage("Performing module initialization..."); + performModuleInitialization(); + + monitor.setMessage("Done initializing"); + } + + /** + * Locate certs file within the Ghidra root directory. If found this will be used + * for initializing the ApplicationTrustManager used for SSL/PKI. + */ + private void locateCACertsFile() { + for (ResourceFile appRoot : Application.getApplicationRootDirectories()) { + File cacertsFile = new File(appRoot.getAbsolutePath(), "cacerts"); + if (cacertsFile.isFile()) { + System.setProperty(ApplicationTrustManagerFactory.GHIDRA_CACERTS_PATH_PROPERTY, + cacertsFile.getAbsolutePath()); + break; + } + } + } + + private void performModuleInitialization() { + List instances = ClassSearcher.getInstances(ModuleInitializer.class); + for (ModuleInitializer initializer : instances) { + monitor.setMessage("Initializing " + initializer.getName() + "..."); + initializer.run(); + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/IterateRepository.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/IterateRepository.java new file mode 100755 index 0000000000..80152eb3a1 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/ingest/IterateRepository.java @@ -0,0 +1,210 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.ingest; + +import java.io.FileNotFoundException; +import java.io.IOException; +import java.net.MalformedURLException; +import java.net.URL; + +import ghidra.features.bsim.query.LSHException; +import ghidra.framework.client.NotConnectedException; +import ghidra.framework.model.DomainFile; +import ghidra.framework.model.DomainFolder; +import ghidra.framework.protocol.ghidra.*; +import ghidra.program.database.ProgramContentHandler; +import ghidra.program.model.listing.Program; +import ghidra.util.Msg; +import ghidra.util.exception.CancelledException; +import ghidra.util.exception.VersionException; +import ghidra.util.task.TaskMonitor; + +public abstract class IterateRepository { + + /** + * Perform processing on program obtained from repository. + * @param program program obtained from repository + * @param monitor processing task monitor + * @throws Exception if an error occured during processing. + * @throws CancelledException if processing was cancelled + */ + protected abstract void process(Program program, TaskMonitor monitor) + throws Exception, CancelledException; + + /** + * Process the specified repository URL + * @param ghidraURL ghidra URL for existing server repository and optional + * folder path + * @param monitor task monitor + * @throws Exception if an error occurs during processing + * @throws CancelledException if processing is cancelled + */ + public void process(URL ghidraURL, TaskMonitor monitor) throws Exception, CancelledException { + + if (!GhidraURL.isServerRepositoryURL(ghidraURL) && + !GhidraURL.isLocalProjectURL(ghidraURL)) { + throw new MalformedURLException("Unsupported repository URL: " + ghidraURL); + } + + URL repoURL = GhidraURL.getProjectURL(ghidraURL); + String path = GhidraURL.getProjectPathname(ghidraURL); + + String finalelement = null; + path = path.trim(); + if (!path.endsWith("/")) { + int pos = path.lastIndexOf('/'); + if (pos >= 0) { + String tmp = path.substring(0, pos + 1); + if (tmp.length() != 0 && !tmp.equals("/")) { + finalelement = path.substring(pos + 1); // A possible file name at the end of the path + path = tmp; + + if (GhidraURL.isServerRepositoryURL(ghidraURL)) { + ghidraURL = new URL(repoURL + path); + } + else { + ghidraURL = new URL(repoURL + "?" + path); + } + } + } + } + + try { + GhidraURLConnection c = (GhidraURLConnection) ghidraURL.openConnection(); + + Msg.debug(IterateRepository.class, "Opening ghidra repository: " + ghidraURL); + Object obj = c.getContent(); + if (!(obj instanceof GhidraURLWrappedContent)) { + throw new IOException("Connect to repository folder failed"); + } + + Object consumer = new Object(); + + GhidraURLWrappedContent wrappedContent = (GhidraURLWrappedContent) obj; + Object content = null; + try { + content = wrappedContent.getContent(consumer); + if (!(content instanceof DomainFolder)) { + throw new IOException("Connect to repository folder failed"); + } + + DomainFolder folder = (DomainFolder) content; + + int totalFiles = getTotalFileCount(folder); + + monitor.setMaximum(totalFiles); + monitor.setShowProgressValue(true); + + if (finalelement != null) { + DomainFolder subfolder = folder.getFolder(finalelement); + + if (subfolder != null) { + folder = subfolder; + // fall thru to the DomainFile and DomainFolder loop + } + else { + DomainFile file = folder.getFile(finalelement); + + if (file == null) { + throw new IOException("Bad folder/file element: " + finalelement); + } + + process(file, monitor); + return; + } + } + + process(folder, monitor); + } + finally { + if (content != null) { + wrappedContent.release(content, consumer); + } + } + } + catch (NotConnectedException e) { + throw new IOException( + "Ghidra repository connection failed (" + repoURL + "): " + e.getMessage()); + } + catch (FileNotFoundException e) { + throw new IOException("Repository path not found: " + path); + } + } + + private void process(DomainFolder folder, TaskMonitor monitor) + throws Exception, CancelledException { + + for (DomainFile file : folder.getFiles()) { + monitor.checkCancelled(); + process(file, monitor); + } + + for (DomainFolder subfolder : folder.getFolders()) { + monitor.checkCancelled(); + process(subfolder, monitor); + } + } + + /** + * Returns the total number of files under the given folder. This does a recursive + * check to search all subdirs. + * + * @param folder the folder to search + * @return total number of files in the folder (and its subfolders) + */ + private int getTotalFileCount(DomainFolder folder) { + int count = 0; + count += folder.getFiles().length; + + for (DomainFolder subfolder : folder.getFolders()) { + count += getTotalFileCount(subfolder); + } + + return count; + } + + private void process(DomainFile file, TaskMonitor monitor) + throws Exception, CancelledException { + + // Do not follow folder-links or consider program links. Using content type + // to filter is best way to control this. If program links should be considered + // "Program.class.isAssignableFrom(domainFile.getDomainObjectClass())" + // should be used. + if (!ProgramContentHandler.PROGRAM_CONTENT_TYPE.equals(file.getContentType())) { + // NOTE: linked-folders and linked-files are not currently supported + return; // skip non-program file + } + + Program program = null; + Object consumer = new Object(); + try { + Msg.debug(IterateRepository.class, "Processing " + file.getPathname() + "..."); + monitor.setMessage("Processing: " + file.getName()); + monitor.incrementProgress(1); + program = (Program) file.getReadOnlyDomainObject(consumer, -1, monitor); + process(program, monitor); + } + catch (VersionException e) { + Msg.error(IterateRepository.class, + "Failed to process file " + file.getPathname() + ": " + e.getMessage()); + } + finally { + if (program != null) { + program.release(consumer); + } + } + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/postgresql/Handler.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/postgresql/Handler.java new file mode 100644 index 0000000000..b8b242d94b --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/postgresql/Handler.java @@ -0,0 +1,49 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.postgresql; + +import java.io.IOException; +import java.net.*; + +/** + * Dummy stream handler, so we can create URL objects with protocol "postgresql" + */ +public class Handler extends URLStreamHandler { + + private static String MY_PARENT_PACKAGE = "ghidra.features.bsim.query"; + private static String PROTOCOL_HANDLER_PKGS_PROPERTY = "java.protocol.handler.pkgs"; + + @Override + protected URLConnection openConnection(URL u) throws IOException { + throw new IOException("Trying to open connection with dummy handler"); + } + + public static void registerHandler() { + String pkgs = System.getProperty(PROTOCOL_HANDLER_PKGS_PROPERTY); + if (pkgs != null) { + if (pkgs.indexOf(MY_PARENT_PACKAGE) >= 0) { + return; // avoid multiple registrations + } + pkgs = pkgs + "|" + MY_PARENT_PACKAGE; + } + else { + pkgs = MY_PARENT_PACKAGE; + } + + System.setProperty(PROTOCOL_HANDLER_PKGS_PROPERTY, pkgs); + + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/AdjustVectorIndex.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/AdjustVectorIndex.java new file mode 100755 index 0000000000..85ef646bcd --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/AdjustVectorIndex.java @@ -0,0 +1,64 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Request that a BSim database either drop or build its main vector index + * + */ +public class AdjustVectorIndex extends BSimQuery { + + public boolean doRebuild; // true if vector index should be rebuilt, false if it should be dropped + public ResponseAdjustIndex adjustresponse; + + public AdjustVectorIndex() { + super("adjustindex"); + doRebuild = false; + } + + /* (non-Javadoc) + * @see ghidra.query.protocol.QueryResponseRecord#buildResponseTemplate() + */ + public void buildResponseTemplate() { + if (response == null) + response = adjustresponse = new ResponseAdjustIndex(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name); + fwrite.append(" rebuild=\""); + fwrite.append(SpecXmlUtils.encodeBoolean(doRebuild)); + fwrite.append("\"/>\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + XmlElement el = parser.start(name); + doRebuild = SpecXmlUtils.decodeBoolean(el.getAttribute("rebuild")); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/BSimFilter.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/BSimFilter.java new file mode 100755 index 0000000000..0577f5e3cb --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/BSimFilter.java @@ -0,0 +1,346 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; +import java.util.Map.Entry; +import java.util.stream.Collectors; + +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.gui.filters.FunctionTagBSimFilterType; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Suitable for client side filtering by calling isFiltered with an ExecutableRecord + * or evaluate with a FunctionDescription. Contains information for passing filter to + * server side. Each 'atom' of the filter (FilterAtom) is expressed as an operator and + * a value string. The operator (FilterType) indicates what part of the ExecutableRecord + * or FunctionDescription must match (or not match) the value string. + */ +public class BSimFilter { + private List atoms; + private int filterflags_mask; // (mask,value) pair for what bits should be + // set/unset in FunctionDescription flags + private int filterflags_value; + + // Cached maps used by the evaluate method + private Map> filterNameToFilterMapAND; + private Map> filterNameToFilterMapOR; + + public BSimFilter() { + atoms = new ArrayList(); + filterflags_mask = 0; + filterflags_value = 0; + filterNameToFilterMapAND = null; + filterNameToFilterMapOR = null; + } + + public int numAtoms() { + return atoms.size(); + } + + public FilterAtom getAtom(int i) { + return atoms.get(i); + } + + public void addAtom(BSimFilterType type, String val) { + if (type.isChildFilter()) { + String exe = "unknown"; + if (val.charAt(0) == '[') { + int i = val.indexOf(']'); + if (i >= 0) { + exe = val.substring(1, i); + val = val.substring(i + 1); + } + } + ChildAtom childatom = new ChildAtom(); + childatom.type = type; + childatom.value = null; + childatom.name = val; + childatom.exename = exe; + atoms.add(childatom); + } + else { + FilterAtom newatom = new FilterAtom(type, val); + if (newatom.isValid()) { + atoms.add(newatom); + if (type instanceof FunctionTagBSimFilterType) { // If this is a function tag filter + int flag = ((FunctionTagBSimFilterType) type).getFlag(); + filterflags_mask |= flag; // Accumulate the mask/value pair here + if (newatom.value.equals("true")) { + filterflags_value |= flag; + } + } + } + } + } + + @Override + public BSimFilter clone() { + BSimFilter op2 = new BSimFilter(); + for (int i = 0; i < atoms.size(); ++i) { + op2.atoms.add(atoms.get(i).clone()); + } + op2.filterflags_mask = filterflags_mask; + op2.filterflags_value = filterflags_value; + return op2; + } + + public void clear() { + atoms.clear(); + filterflags_mask = 0; + filterflags_value = 0; + } + + public boolean isEmpty() { + if (filterflags_mask != 0) { + return false; + } + for (int i = 0; i < atoms.size(); ++i) { + if (!atoms.get(i).type.isBlank()) { + return false; + } + } + return true; + } + + public void saveXml(Writer fwrite) throws IOException { + fwrite.append(""); + if (filterflags_mask != 0) { + fwrite.append(""); + fwrite.append(SpecXmlUtils.encodeUnsignedInteger(filterflags_value)); + fwrite.append("\n"); + } + for (int i = 0; i < atoms.size(); ++i) { + atoms.get(i).saveXml(fwrite); + } + fwrite.append("\n"); + } + + public void restoreXml(XmlPullParser parser) { + parser.start("exefilter"); + atoms.clear(); + while (parser.peek().isStart()) { + XmlElement el = parser.peek(); + if (el.getName().equals("flags")) { + el = parser.start(); + filterflags_mask = SpecXmlUtils.decodeInt(el.getAttribute("mask")); + filterflags_value = SpecXmlUtils.decodeInt(parser.end().getText()); + } + else if (el.getName().equals("childatom")) { + ChildAtom newatom = new ChildAtom(); + newatom.restoreXml(parser); + atoms.add(newatom); + } + else { + FilterAtom newatom = new FilterAtom(); + newatom.restoreXml(parser); + atoms.add(newatom); + } + } + parser.end(); + } + + /** + * Returns true if all filters resolve correctly for the given function description. There are + * 4 main types of filters, each of which must be evaluated differently: + * + * 1) Positive Filter: " matches ". + * For these, filter out any result that does not contain all elements (at a minimum) of the + * filter value. + * ie: FILTER = "SetA", RESULT = "SetA" => keep it + * FILTER = "SetA, SetB", RESULT = "SetA" => filter out + * + * 2) Negative Filter: " does not match " + * For these, filter out any result that does not contain EXACTLY the filter value. + * ie: FILTER = "SetA", RESULT = "SetA, SetB" => keep it + * FILTER = "SetA, SetB", RESULT = "SetA, SetB" => filter out + * + * 3) Positive Exe Filter: Same as #1, but custom exe filters are stored differently than + * 'normal' categories and must be processed separately. + * + * 4) Negative Exe Filter: Same as #2, but custom exe filters are stored differently than + * 'normal' categories and must be processed separately. + * + * @param func the function description + * @return true if all filters resolve to true + */ + public boolean evaluate(FunctionDescription func) { + + if ((func.getFlags() & filterflags_mask) != filterflags_value) { + return false; + } + + ExecutableRecord exe = func.getExecutableRecord(); + + if (filterNameToFilterMapAND == null) { + populateFilterMaps(); + } + + return processFilters(exe); + } + + /** + * Sets up the filterNameToFilter... maps with the appropriate category/filter values. This is done to + * keep all the filters of the same type in the same place. + * + * ie: If one of the filters set is the "Executable name does not equal" filter, and gives it + * two values: "dexdump" and "stty", then the "filterNameToFilterMapOR" map will + * have the following: + * key: "Executable name does not equal" + * value: [dexdump, stty] + * + */ + private void populateFilterMaps() { + // First set up maps to organize which filters should be AND'd together and + // which ones should be OR'd. + filterNameToFilterMapAND = new HashMap<>(); + filterNameToFilterMapOR = new HashMap<>(); + + for (FilterAtom atom : atoms) { + + // The name of the filter will be the map key (ie: "executable name equals"), + // so grab it here. + String name = atom.type.getLabel(); + + BSimFilterType filter = atom.type; + + if (filter.orMultipleEntries()) { + if (!filterNameToFilterMapOR.containsKey(name)) { + List list = new ArrayList(); + list.add(atom); + filterNameToFilterMapOR.put(name, list); + } + else { + filterNameToFilterMapOR.get(name).add(atom); + } + } + else { + if (!filterNameToFilterMapAND.containsKey(name)) { + List list = new ArrayList(); + list.add(atom); + filterNameToFilterMapAND.put(name, list); + } + else { + filterNameToFilterMapAND.get(name).add(atom); + } + } + } + } + + /** + * Takes all the entries in the 4 filter maps and uses them to determine which rows + * should be kept and which should be filtered out. + * + * @param exe the executable record + * @return true if the record should be kept, false if not + */ + private boolean processFilters(ExecutableRecord exe) { + + // Check the standard positive filters ("does match") + for (Map.Entry> entry : filterNameToFilterMapAND.entrySet()) { + List value = entry.getValue(); + if (!evaluateAND(value, exe)) { + return false; + } + } + + // Check the standard negative filters ("does not match") + for (Map.Entry> entry : filterNameToFilterMapOR.entrySet()) { + List value = entry.getValue(); + if (!evaluateOR(value, exe)) { + return false; + } + } + + // If we're here, then it passed all tests, so return true. + return true; + } + + /** + * Return true only if ALL filters evaluate to true. + * + * @param filters the list of all filters + * @param exe the executable record + * @return true if all filters evaluate to true + */ + private boolean evaluateAND(List filters, ExecutableRecord exe) { + + for (FilterAtom filter : filters) { + if (!filter.evaluate(exe)) { + return false; + } + } + + return true; + } + + /** + * Return true if any ONE of the atoms evaluates to true. + * + * @param filters the list of all filters + * @param exe the executable record + * @return true if all filters evaluate to true + */ + private boolean evaluateOR(List filters, ExecutableRecord exe) { + + for (FilterAtom filter : filters) { + if (filter.evaluate(exe)) { + return true; + } + } + + return false; + } + + public void replaceWith(BSimFilter other) { + this.atoms = other.atoms; + this.filterflags_mask = other.filterflags_mask; + this.filterflags_value = other.filterflags_value; + } + + public List getFilterEntries() { + List filterStrings = new ArrayList<>(); + if (filterNameToFilterMapAND == null) { + populateFilterMaps(); + } + for (Entry> entry : filterNameToFilterMapOR.entrySet()) { + List atomList = entry.getValue(); + filterStrings.add(new FilterEntry(atomList.get(0).type, getValues(atomList))); + + } + for (Entry> entry : filterNameToFilterMapAND.entrySet()) { + List atomList = entry.getValue(); + filterStrings.add(new FilterEntry(atomList.get(0).type, getValues(atomList))); + } + return filterStrings; + } + + private List getValues(List atomList) { + return atomList.stream().map(a -> a.getValueString()).collect(Collectors.toList()); + } + + public record FilterEntry(BSimFilterType filterType, List values) {/**/} + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/BSimQuery.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/BSimQuery.java new file mode 100644 index 0000000000..40e0d0fb20 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/BSimQuery.java @@ -0,0 +1,159 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.xml.XmlPullParser; + +// A database query that can be serialized + +/** + * {@link BSimQuery} facilitates all BSim {@link FunctionDatabase} queries + * which when executed provide a specific {@link QueryResponseRecord}. + * + * @param The {@link QueryResponseRecord} response implementation class + */ +public abstract class BSimQuery { + + // TODO: direct manipulation of instance fields for all implementations + // should be utilize tailored constructor arguments + + // TODO: restoreXml method should be replaced by Xml-based constructor for each implementation + + protected final String name; + protected R response; + + public BSimQuery(String name) { + this.name = name; + response = null; + } + + /** + * Executes this query via the {@link FunctionDatabase#query(BSimQuery)} method. + * The use of this method is preferred due to its type enforcement on the returned + * response object. + * @param database BSim function database to be queried + * @return query response or null on error (see {@link FunctionDatabase#getLastError()}). + */ + @SuppressWarnings("unchecked") + public final R execute(FunctionDatabase database) { + return (R) database.query(this); + } + + public void clearResponse() { + response = null; + } + + public R getResponse() { + return response; + } + + public String getName() { + return name; + } + + public void buildResponseTemplate() { + // Any response subclass doesn't need to implement this + } + + public abstract void saveXml(Writer fwrite) throws IOException; + + public abstract void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException; + + public DescriptionManager getDescriptionManager() { return null; } + + /** + * @return a partial clone of this query suitable for holding local stages of the query via StagingManager + */ + public BSimQuery getLocalStagingCopy() { + return null; + } + + /** + * Restore a query from a stream + * @param parser is the XmlPullParser already queued up with the stream to process + * @param vectorFactory is used to generate any vector objects from the XML + * @return one of the Query* instances derived from QueryResponseRecord + * @throws LSHException for errors creating the command + */ + public static BSimQuery restoreQuery(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + String mainName = parser.peek().getName(); + BSimQuery query; + if (mainName.equals("querynearest")) { + query = new QueryNearest(); + } + else if (mainName.equals("querynearestvector")) { + query = new QueryNearestVector(); + } + else if (mainName.equals("insert")) { + query = new InsertRequest(); + } + else if (mainName.equals("queryinfo")) { + query = new QueryInfo(); + } + else if (mainName.equals("update")) { + query = new QueryUpdate(); + } + else if (mainName.equals("queryname")) { + query = new QueryName(); + } + else if (mainName.equals("delete")) { + query = new QueryDelete(); + } + else if (mainName.equals("createdatabase")) { + query = new CreateDatabase(); + } + else if (mainName.equals("querychildren")) { + query = new QueryChildren(); + } + else if (mainName.equals("querycluster")) { + query = new QueryCluster(); + } + else if (mainName.equals("querypair")) { + query = new QueryPair(); + } + else if (mainName.equals("installcategory")) { + query = new InstallCategoryRequest(); + } + else if (mainName.equals("installmetadata")) { + query = new InstallMetadataRequest(); + } + else if (mainName.equals("installtag")) { + query = new InstallTagRequest(); + } + else if (mainName.equals("adjustindex")) { + query = new AdjustVectorIndex(); + } + else if (mainName.equals("passwordchange")) { + query = new PasswordChange(); + } + else if (mainName.equals("prewarmrequest")) { + query = new PrewarmRequest(); + } + else { + throw new LSHException("Unknown query tag: "+mainName); + } + query.restoreXml(parser,vectorFactory); + return query; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ChildAtom.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ChildAtom.java new file mode 100755 index 0000000000..dc1c592062 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ChildAtom.java @@ -0,0 +1,78 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +public class ChildAtom extends FilterAtom { + public String name = null; // Name of the child function + public String exename = null; // Name of the executable (or library) containing the child (or null) + + public void saveXml(Writer fwrite) throws IOException { + fwrite.append("'); + SpecXmlUtils.xmlEscapeWriter(fwrite, name); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser) { + XmlElement el = parser.start("childatom"); + type = BSimFilterType.nameToType(el); + value = null; + exename = el.getAttribute("exe"); + name = parser.end(el).getText(); + } + + public FilterAtom clone() { + ChildAtom newatom = new ChildAtom(); + newatom.type = type; + newatom.value = value; + newatom.name = name; + newatom.exename = exename; + return newatom; + } + + public String getInfoString() { + if (name == null) + return null; + String res = "Has child "; + if (exename != null) + res += '[' + exename + ']'; + res += name; + return res; + } + + @Override + public String getValueString() { + if (exename != null) { + return '[' + exename + ']' + name; + } + return name; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ClusterNote.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ClusterNote.java new file mode 100755 index 0000000000..20f91291d7 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ClusterNote.java @@ -0,0 +1,76 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.Map; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +public class ClusterNote { + private FunctionDescription func; + private int setsize; // Number of hits in cluster + private double maxscore; // Highest similarity + private double signif; // Significance of highest similarity + + public ClusterNote() {} // For use with restoreXml + + public ClusterNote(FunctionDescription f,int ss,double ms,double sig) { + func = f; + setsize = ss; + maxscore = ms; + signif = sig; + } + + public FunctionDescription getFunctionDescription() { return func; } + + public double getMaxSimilarity() { return maxscore; } + public double getSignificance() { return signif; } + + public void saveXml(Writer write) throws IOException { + StringBuilder buf = new StringBuilder(); + buf.append("\n"); + buf.append(" ").append(SpecXmlUtils.encodeSignedInteger(setsize)).append("\n"); + buf.append(" ").append(Double.toString(maxscore)).append("\n"); + buf.append(" ").append(Double.toString(signif)).append("\n"); + buf.append("\n"); + write.append(buf.toString()); + } + + public void restoreXml(XmlPullParser parser,DescriptionManager manage,Map xrefMap) throws LSHException { + XmlElement el = parser.start("note"); + int id = SpecXmlUtils.decodeInt(el.getAttribute("id")); + ExecutableRecord exe = xrefMap.get(id); + long address = SpecXmlUtils.decodeLong(el.getAttribute("addr")); + func = manage.findFunction(el.getAttribute("name"), address, exe); + parser.start("setsize"); + setsize = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("sim"); + maxscore = Double.parseDouble(parser.end().getText()); + parser.start("sig"); + signif = Double.parseDouble(parser.end().getText()); + parser.end(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/CreateDatabase.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/CreateDatabase.java new file mode 100755 index 0000000000..f35a28c278 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/CreateDatabase.java @@ -0,0 +1,59 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +public class CreateDatabase extends BSimQuery { + public String config_template; // Name of configuration to use for database + public DatabaseInformation info; // Some overrides for the configuration + public ResponseInfo inforesponse; + + public CreateDatabase() { + super("createdatabase"); + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = inforesponse = new ResponseInfo(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name); + fwrite.append(" template=\"").append(config_template).append("\">\n"); + info.saveXml(fwrite); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + XmlElement el = parser.start(name); + config_template = el.getAttribute("template"); + info = new DatabaseInformation(); + info.restoreXml(parser); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ExeSpecifier.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ExeSpecifier.java new file mode 100755 index 0000000000..f5922c2e34 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ExeSpecifier.java @@ -0,0 +1,140 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.*; + +import org.apache.commons.lang3.StringUtils; + +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +public class ExeSpecifier implements Comparable { + + public String exename = ""; + public String arch = ""; + public String execompname = ""; + public String exemd5 = ""; + + public void saveXml(Writer fwrite) throws IOException { + fwrite.append(" \n"); + if (exemd5.length() != 0) { + fwrite.append(" ").append(exemd5).append("\n"); + fwrite.append(" "); + if (exename.length() != 0) { + SpecXmlUtils.xmlEscapeWriter(fwrite, exename); + } + fwrite.append("\n"); + } + else { + fwrite.append(" "); + if (exename.length() != 0) { + SpecXmlUtils.xmlEscapeWriter(fwrite, exename); + } + fwrite.append("\n"); + fwrite.append(" "); + if (arch.length() != 0) { + SpecXmlUtils.xmlEscapeWriter(fwrite, arch); + } + fwrite.append("\n"); + fwrite.append(" "); + if (execompname.length() != 0) { + SpecXmlUtils.xmlEscapeWriter(fwrite, execompname); + } + fwrite.append("\n"); + } + fwrite.append(" \n"); + } + + public void restoreXml(XmlPullParser parser) { + parser.start(); + XmlElement el = parser.start(); + if (el.getName().equals("md5")) { + exemd5 = parser.end().getText(); + parser.start(); + exename = parser.end().getText(); + arch = ""; + execompname = ""; + } + else { + exemd5 = ""; + exename = parser.end().getText(); + parser.start("arch"); + arch = parser.end().getText(); + parser.start("compiler"); + execompname = parser.end().getText(); + } + parser.end(); + } + + public void transfer(ExecutableRecord op2) { + exemd5 = op2.getMd5(); + exename = op2.getNameExec(); + arch = ""; + execompname = ""; + } + + public String getExeNameWithMD5() { + StringBuilder buf = new StringBuilder(); + boolean addspace = false; + if (!StringUtils.isBlank(exename)) { + buf.append(exename); + addspace = true; + } + if (!StringUtils.isBlank(exemd5)) { + if (addspace) { + buf.append(' '); + } + buf.append(exemd5); + } + return buf.toString(); + } + + @Override + public boolean equals(Object obj) { + ExeSpecifier o = (ExeSpecifier) obj; + if (exemd5.length() != 0) { + return exemd5.equals(o.exemd5); + } + boolean cmp = exename.equals(o.exename); + if (!cmp) { + return false; + } + cmp = arch.equals(o.arch); + if (!cmp) { + return false; + } + return execompname.equals(o.execompname); + } + + @Override + public int compareTo(ExeSpecifier o) { + if (exemd5.length() != 0) { + return exemd5.compareTo(o.exemd5); + } + int comp = exename.compareTo(o.exename); + if (comp != 0) { + return comp; + } + comp = arch.compareTo(o.arch); + if (comp != 0) { + return comp; + } + return execompname.compareTo(o.execompname); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ExecutableResultWithDeDuping.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ExecutableResultWithDeDuping.java new file mode 100755 index 0000000000..8ec0e9d1df --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ExecutableResultWithDeDuping.java @@ -0,0 +1,137 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.util.*; + +import ghidra.features.bsim.query.description.*; + +public class ExecutableResultWithDeDuping implements Comparable { + private RowKey id; // Copy ExecutableRecord id + private ExecutableRecord exerecord; + private int funccount; // Number of functions matching into this executable + private double sumsignif; // Sum of all matching function significance + + private ExecutableResultWithDeDuping() { + id = null; + exerecord = null; + funccount = 0; + sumsignif = 0.0; + } + + public ExecutableResultWithDeDuping(ExecutableRecord rec) { + exerecord = rec; + id = rec.getRowId(); + funccount = 0; + sumsignif = 0.0; + } + + public ExecutableRecord getExecutableRecord() { + return exerecord; + } + + public void addFunction(double signif) { + funccount += 1; + sumsignif += signif; + } + + /** + * @return number of functions with matches into this executable + */ + public int getFunctionCount() { + return funccount; + } + + /** + * @return sum of significance scores for all matching functions + */ + public double getSignificanceSum() { + return sumsignif; + } + + @Override + public boolean equals(Object obj) { + return (id.equals( ((ExecutableResultWithDeDuping)obj).id) ); + } + + @Override + /*public int compareTo(ExecutableResultWithDeDuping o) { + long id2 = o.id; + if (id == id2) return 0; + return (id < id2) ? -1 : 1; + }*/ + + public int compareTo(ExecutableResultWithDeDuping o){ + return Double.compare(this.getSignificanceSum(), o.getSignificanceSum()); + } + + + + public static Collection generate(Iterator iter, Map duplicationInfo) { + TreeSet res = new TreeSet(); + ExecutableResultWithDeDuping curres = new ExecutableResultWithDeDuping(); + while(iter.hasNext()) { + SimilarityResult simres = iter.next(); + TreeSet exetree = new TreeSet(); + Iterator noteiter = simres.iterator(); + + Integer totalNumDuplicates = duplicationInfo.get(simres.getBase()); //should never be null + if(totalNumDuplicates == null){ + totalNumDuplicates = 1000; + } + Map dupesInExecutable = new HashMap(); + + while(noteiter.hasNext()) { + SimilarityNote note = noteiter.next(); + curres.exerecord = note.getFunctionDescription().getExecutableRecord(); + curres.id = curres.exerecord.getRowId(); + ExecutableResultWithDeDuping tmpres = exetree.floor(curres); + if ((tmpres == null)||(!tmpres.id.equals(curres.id))) { // Haven't seen this executable before + tmpres = new ExecutableResultWithDeDuping(curres.exerecord); + exetree.add(tmpres); + tmpres.sumsignif = note.getSignificance(); + dupesInExecutable.put(tmpres, 1); + } + else { // Seen this executable before for this SimilarityResult + + if (tmpres.sumsignif < note.getSignificance()){ + tmpres.sumsignif = note.getSignificance(); // Find maximum significance result for this executable + dupesInExecutable.put(tmpres, 1); //found a higher significance match - reset the count + } + else{ + if( (tmpres.sumsignif == note.getSignificance()) && (dupesInExecutable.get(tmpres) < totalNumDuplicates)){ + tmpres.sumsignif = tmpres.sumsignif + note.getSignificance(); + dupesInExecutable.put(tmpres, dupesInExecutable.get(tmpres) + 1); //increment the count + } + + } + } + } + + Iterator finaliter = exetree.iterator(); + while(finaliter.hasNext()) { + ExecutableResultWithDeDuping eres = finaliter.next(); + ExecutableResultWithDeDuping tmpres = res.floor(eres); + if ((tmpres == null)||(!tmpres.id.equals(eres.id))) { + tmpres = new ExecutableResultWithDeDuping(eres.exerecord); + res.add(tmpres); + } + tmpres.addFunction(eres.sumsignif); + } + } + return res; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FilterAtom.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FilterAtom.java new file mode 100755 index 0000000000..e4d991cc84 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FilterAtom.java @@ -0,0 +1,103 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import ghidra.features.bsim.gui.filters.BSimFilterType; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * A single element for filtering on specific properties of ExecutableRecords or FunctionDescriptions + * Each FilterAtom consists of a FilterTemplate describing the property to filter on, and how the filter should apply, + * and a String -value- that the property should match (or not) + * + */ +public class FilterAtom { + public BSimFilterType type; // Type of filter to perform + public String value; // Constant data to use in the filter + + public FilterAtom() { + + } + + public FilterAtom(BSimFilterType type, String value) { + this.type = type; + this.value = type.normalizeValue(value); + } + + @Override + public FilterAtom clone() { + FilterAtom res = new FilterAtom(); + res.type = type; + res.value = value; + return res; + } + + public void saveXml(Writer fwrite) throws IOException { + fwrite.append("'); + SpecXmlUtils.xmlEscapeWriter(fwrite, value); + fwrite.append("\n"); + } + + public void restoreXml(XmlPullParser parser) { + XmlElement el = parser.start("atom"); + type = BSimFilterType.nameToType(el); + value = parser.end().getText(); + if (type.isValidValue(value)) { + value = type.normalizeValue(value); + } + else { + type = BSimFilterType.getBlank(); + } + } + + public String getInfoString() { + if (type.isBlank()) { + return null; + } + String res = type.toString() + ' ' + value; + return res; + } + + /** + * @param rec is a specific ExecutableRecord + * @return true if this FilterAtom would let the specific executable pass the filter + */ + public boolean evaluate(ExecutableRecord rec) { + if (value == null) { + return true; + } + return type.evaluate(rec, value); + } + + /** + * Returns true if this Atom has a non-null value + */ + public boolean isValid() { + return value != null; + } + + public String getValueString() { + return value; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FunctionEntry.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FunctionEntry.java new file mode 100755 index 0000000000..49bf806db3 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FunctionEntry.java @@ -0,0 +1,59 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Identifying information for a function within a single executable + * + */ +public class FunctionEntry { + public String funcName; // Name of the function within the executable + public long address; // Address of the function + + private FunctionEntry() { + // Used for restoreXml + } + + public FunctionEntry(FunctionDescription desc) { + funcName = desc.getFunctionName(); + address = desc.getAddress(); + } + + public void saveXml(Writer writer) throws IOException { + writer.append("\n"); + } + + public static FunctionEntry restoreXml(XmlPullParser parser) { + FunctionEntry functionEntry = new FunctionEntry(); + XmlElement startEl = parser.start("fentry"); + functionEntry.funcName = startEl.getAttribute("name"); + functionEntry.address = SpecXmlUtils.decodeLong(startEl.getAttribute("addr")); + parser.end(startEl); + return functionEntry; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FunctionStaging.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FunctionStaging.java new file mode 100755 index 0000000000..3083d3da1e --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/FunctionStaging.java @@ -0,0 +1,78 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.util.Iterator; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.features.bsim.query.description.FunctionDescription; + +public class FunctionStaging extends StagingManager { + private BSimQuery localQuery; + private int stagesize; // Number of functions per stage + private Iterator curiter; + private DescriptionManager gmanage; // The global function manager + private DescriptionManager imanage; // The internal function manager + + public FunctionStaging(int stagesize) { + this.stagesize = stagesize; + localQuery = null; + } + + @Override + public BSimQuery getQuery() { + return localQuery; + } + + @Override + public boolean initialize(BSimQuery q) throws LSHException { + globalQuery = q; + gmanage = q.getDescriptionManager(); + if (gmanage == null) + throw new LSHException("Query cannot be function staged"); + totalsize = gmanage.numFunctions(); + queriesmade = 0; + localQuery = q.getLocalStagingCopy(); + imanage = localQuery.getDescriptionManager(); + + curiter = gmanage.listAllFunctions(); + imanage.clear(); + imanage.transferSettings(gmanage); + int count; + for (count = 0; count < stagesize; ++count) { + if (!curiter.hasNext()) + break; + imanage.transferFunction(curiter.next(), true); // Copy the next function into manager for this stage + queriesmade += 1; + } + return (count != 0); + } + + @Override + public boolean nextStage() throws LSHException { + imanage.clear(); + imanage.transferSettings(gmanage); + int count; + for (count = 0; count < stagesize; ++count) { + if (!curiter.hasNext()) + break; + imanage.transferFunction(curiter.next(), true); + queriesmade += 1; + } + return (count != 0); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InsertOptionalValues.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InsertOptionalValues.java new file mode 100755 index 0000000000..dc7077f133 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InsertOptionalValues.java @@ -0,0 +1,116 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Insert key/value pairs into an optional table + */ +public class InsertOptionalValues extends BSimQuery { + + public ResponseOptionalExist optionalresponse = null; + public String tableName; // Name of optional SQL table + public int keyType; // type-code of key as per java.sql.Types + public int valueType; // type-code of value + public Object[] keys; // keys to be inserted + public Object[] values; // values (corresponding to keys) to be inserted + + public InsertOptionalValues() { + super("insertoptionalvalues"); + tableName = null; + keyType = -1; + valueType = -1; + keys = null; + values = null; + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = optionalresponse = new ResponseOptionalExist(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + fwrite.append(""); + SpecXmlUtils.xmlEscapeWriter(fwrite, tableName); + fwrite.append("\n"); + fwrite.append(""); + fwrite.append(Integer.toString(keyType)); + fwrite.append("\n"); + fwrite.append(""); + fwrite.append(Integer.toString(valueType)); + fwrite.append("\n"); + for (Object key : keys) { + fwrite.append(""); + SpecXmlUtils.xmlEscapeWriter(fwrite, key.toString()); + fwrite.append("\n"); + } + for (Object val : values) { + fwrite.append(""); + SpecXmlUtils.xmlEscapeWriter(fwrite, val.toString()); + fwrite.append("\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + keys = null; + values = null; + List resultKeys = new ArrayList(); + List resultValues = new ArrayList(); + parser.start(name); + parser.start("tablename"); + tableName = parser.end().getText(); + parser.start("keytype"); + keyType = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("valuetype"); + valueType = SpecXmlUtils.decodeInt(parser.end().getText()); + while (parser.peek().isStart()) { + XmlElement el = parser.start(); + String nm = el.getName(); + String body = parser.end().getText(); + if (nm.equals("key")) { + resultKeys.add(body); + } + else { + resultValues.add(body); + } + } + parser.end(); + if (!resultKeys.isEmpty()) { + keys = new Object[resultKeys.size()]; + values = new Object[resultValues.size()]; + resultKeys.toArray(keys); + resultValues.toArray(values); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InsertRequest.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InsertRequest.java new file mode 100755 index 0000000000..4cef4894ba --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InsertRequest.java @@ -0,0 +1,95 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Request that specific executables and functions (as described by ExecutableRecords and FunctionDescriptions) + * by inserted into a BSim database. + * + */ +public class InsertRequest extends BSimQuery { + + public DescriptionManager manage; // The set of executables and functions to be inserted + public String repo_override; // Override of repository for this insert + public String path_override; // Override of path + public ResponseInsert insertresponse; + + public InsertRequest() { + super("insert"); + manage = new DescriptionManager(); + repo_override = null; + path_override = null; + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = insertresponse = new ResponseInsert(); + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public InsertRequest getLocalStagingCopy() { + InsertRequest newi = new InsertRequest(); + newi.repo_override = repo_override; + newi.path_override = path_override; + return newi; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + manage.saveXml(fwrite); + if (repo_override != null) + fwrite.append("").append(repo_override).append("\n"); + if (path_override != null) + fwrite.append("").append(path_override).append("\n"); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser,LSHVectorFactory vectorFactory) throws LSHException { + parser.start(name); + manage.restoreXml(parser,vectorFactory); + XmlElement subel = parser.peek(); + while(subel.isStart()) { + if (subel.getName().equals("repository")) { // Optional repository + parser.start("repository"); + repo_override = parser.end().getText(); + } + else if (subel.getName().equals("path")) { // Optional path + parser.start("path"); + path_override = parser.end().getText(); + } + subel = parser.peek(); + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallCategoryRequest.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallCategoryRequest.java new file mode 100755 index 0000000000..9afe4e908a --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallCategoryRequest.java @@ -0,0 +1,69 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.CategoryRecord; +import ghidra.util.xml.XmlUtilities; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Request that a new executable category be installed for a specific BSim server. + * + */ +public class InstallCategoryRequest extends BSimQuery { + + public String type_name; // Name of new type of category + public boolean isdatecolumn; // True if name should be treated as new name for date column + public ResponseInfo installresponse; + + public InstallCategoryRequest() { + super("installcategory"); + type_name = ""; + isdatecolumn = false; + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = installresponse = new ResponseInfo(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + if (!CategoryRecord.enforceTypeCharacters(type_name)) + throw new IOException("Bad characters in requested category type"); + fwrite.append('<').append(name); + if (isdatecolumn) + fwrite.append(" datecolumn=\"true\""); + fwrite.append('>'); + fwrite.append(type_name); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + XmlElement el = parser.start(name); + isdatecolumn = XmlUtilities.parseBoolean(el.getAttribute("datecolumn")); + type_name = parser.end().getText(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallMetadataRequest.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallMetadataRequest.java new file mode 100755 index 0000000000..25b1c3a672 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallMetadataRequest.java @@ -0,0 +1,82 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.XmlUtilities; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Request that the high-level meta-data fields (name,owner,description) of a database be changed + * + */ +public class InstallMetadataRequest extends BSimQuery { + + public String dbname; // New name of database (if null, old value will be retained) + public String owner; // New owner of database (if null, old value will be retained) + public String description; // New description for database (if null, old value will be retained) + + public ResponseInfo installresponse; + + public InstallMetadataRequest() { + super("installmetadata"); + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = installresponse = new ResponseInfo(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + if (dbname!=null && dbname.length()!=0) + fwrite.append("").append(XmlUtilities.escapeElementEntities(dbname)).append("\n"); + if (owner!=null && owner.length()!=0) + fwrite.append("").append(XmlUtilities.escapeElementEntities(owner)).append("\n"); + if (description!=null && description.length()!=0) + fwrite.append("").append(XmlUtilities.escapeElementEntities(description)).append("\n"); + fwrite.append("\n"); + } + + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + dbname = null; + owner = null; + description = null; + parser.start(name); + while(parser.peek().isStart()) { + XmlElement start = parser.start(); + String elname = start.getName(); + String val = parser.end().getText(); + if (elname.equals("name")) + dbname = val; + else if (elname.equals("owner")) + owner = val; + else if (elname.equals("description")) + description = val; + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallTagRequest.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallTagRequest.java new file mode 100755 index 0000000000..1419da1bff --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/InstallTagRequest.java @@ -0,0 +1,65 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.CategoryRecord; +import ghidra.xml.XmlPullParser; + +/** + * Request that a new function tag be installed for a specific BSim server + * + */ +public class InstallTagRequest extends BSimQuery { + + public String tag_name; // Name of new function tag + + public ResponseInfo installresponse; + + public InstallTagRequest() { + super("installtag"); + tag_name = ""; + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = installresponse = new ResponseInfo(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + if (!CategoryRecord.enforceTypeCharacters(tag_name)) { + throw new IOException("Bad characters in requested category type"); + } + fwrite.append('<').append(name); + fwrite.append('>'); + fwrite.append(tag_name); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + parser.start(name); + tag_name = parser.end().getText(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/NullStaging.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/NullStaging.java new file mode 100755 index 0000000000..5de835f9ee --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/NullStaging.java @@ -0,0 +1,47 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; + +public class NullStaging extends StagingManager { + + @Override + public BSimQuery getQuery() { + return globalQuery; + } + + @Override + public boolean initialize(BSimQuery q) throws LSHException { + globalQuery = q; + totalsize = 0; + queriesmade = 0; + + DescriptionManager imanage = q.getDescriptionManager(); + if (imanage == null) + return true; + + totalsize = imanage.numFunctions(); + return (totalsize != 0); // Is there any data at all for an initial stage + } + + @Override + public boolean nextStage() { + queriesmade = totalsize; + return false; // There is always only one stage + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PairInput.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PairInput.java new file mode 100755 index 0000000000..fa277bcfe4 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PairInput.java @@ -0,0 +1,53 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Identifiers for a pair of functions + * + */ +public class PairInput { + public ExeSpecifier execA; + public FunctionEntry funcA; + public ExeSpecifier execB; + public FunctionEntry funcB; + + public void saveXml(Writer writer) throws IOException { + writer.append("\n"); + execA.saveXml(writer); + funcA.saveXml(writer); + execB.saveXml(writer); + funcB.saveXml(writer); + writer.append("\n"); + } + + public void restoreXml(XmlPullParser parser) { + XmlElement startEl = parser.start("pair"); + execA = new ExeSpecifier(); + execA.restoreXml(parser); + funcA = FunctionEntry.restoreXml(parser); + execB = new ExeSpecifier(); + execB.restoreXml(parser); + funcB = FunctionEntry.restoreXml(parser); + parser.end(startEl); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PairNote.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PairNote.java new file mode 100755 index 0000000000..89748d69cb --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PairNote.java @@ -0,0 +1,122 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Result of a comparison between two functions. + * Includes descriptors for the original functions, the similarity and significance scores + * and other score information. + * + */ +public class PairNote { + private ExeSpecifier exe1; + private FunctionEntry func1; // First function + private ExeSpecifier exe2; + private FunctionEntry func2; // Second function + private double sim; // Similarity + private double signif; // Significance + private double dotprod; // Unnormalized dot product + private int count1; // Number of hashes from func1 + private int count2; // Number of hashes from func2 + private int icount; // Number of hashes in intersection + + public double getSimilarity() { + return sim; + } + + public double getSignificance() { + return signif; + } + + public double getDotProduct() { + return dotprod; + } + + public int getFunc1HashCount() { + return count1; + } + + public int getFunc2HashCount() { + return count2; + } + + public int getIntersectionCount() { + return icount; + } + + public PairNote() { // For use with restoreXml + } + + public PairNote(FunctionDescription f1,FunctionDescription f2,double sm,double sf,double dp, + int c1,int c2,int ic) { + exe1 = new ExeSpecifier(); + exe1.transfer(f1.getExecutableRecord()); + func1 = new FunctionEntry(f1); + exe2 = new ExeSpecifier(); + exe2.transfer(f2.getExecutableRecord()); + func2 = new FunctionEntry(f2); + sim = sm; + signif = sf; + dotprod = dp; + count1 = c1; + count2 = c2; + icount = ic; + } + + public void saveXml(Writer writer) throws IOException { + writer.append("\n"); + exe1.saveXml(writer); + func1.saveXml(writer); + exe2.saveXml(writer); + func2.saveXml(writer); + writer.append(" ").append(Double.toString(sim)).append("\n"); + writer.append(" ").append(Double.toString(signif)).append("\n"); + writer.append(" ").append(Double.toString(dotprod)).append("\n"); + writer.append(" ").append(Integer.toString(count1)).append("\n"); + writer.append(" ").append(Integer.toString(count2)).append("\n"); + writer.append(" ").append(Integer.toString(icount)).append("\n"); + writer.append("\n"); + } + + public void restoreXml(XmlPullParser parser) { + XmlElement startEl = parser.start("note"); + exe1 = new ExeSpecifier(); + exe1.restoreXml(parser); + func1 = FunctionEntry.restoreXml(parser); + exe2 = new ExeSpecifier(); + exe2.restoreXml(parser); + func2 = FunctionEntry.restoreXml(parser); + parser.start("sim"); + sim = Double.parseDouble(parser.end().getText()); + parser.start("sig"); + signif = Double.parseDouble(parser.end().getText()); + parser.start("dot"); + dotprod = Double.parseDouble(parser.end().getText()); + parser.start("cnt1"); + count1 = SpecXmlUtils.decodeInt(parser.end().getText()); + count2 = SpecXmlUtils.decodeInt(parser.end().getText()); + icount = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.end(startEl); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PasswordChange.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PasswordChange.java new file mode 100755 index 0000000000..38290d42b0 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PasswordChange.java @@ -0,0 +1,80 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Request a password change for a specific user + * Currently provides no explicit protection for password data on the client. + * Should be used in conjunction with connection encryption (SSL) to protect + * data in transit to the server. + */ +public class PasswordChange extends BSimQuery { + + public ResponsePassword passwordResponse; + public String username; // Identifier for user whose password should be changed + public char[] newPassword; // The new password as raw character data + + public PasswordChange() { + super("passwordchange"); + username = null; + newPassword = null; + } + + /** + * Clear the password data. (Should be) used by database client immediately upon sending request to server + */ + public void clearPassword() { + if (newPassword != null) { + for (int i = 0; i < newPassword.length; ++i) { + newPassword[i] = ' '; + } + } + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = passwordResponse = new ResponsePassword(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append("<").append(name); + fwrite.append(" username=\"").append(username); + fwrite.append("\">"); + SpecXmlUtils.xmlEscapeWriter(fwrite, new String(newPassword)); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + XmlElement el = parser.start(name); + username = el.getAttribute("username"); + newPassword = parser.end().getText().toCharArray(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PreFilter.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PreFilter.java new file mode 100755 index 0000000000..ca9453a503 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PreFilter.java @@ -0,0 +1,48 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.util.ArrayList; +import java.util.List; +import java.util.function.BiPredicate; + +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.program.model.listing.Program; + +public class PreFilter { + private List> preFilters; + + public PreFilter() { + preFilters = new ArrayList<>(); + } + + public void addPredicate(BiPredicate predicate) { + preFilters.add(predicate); + } + + public BiPredicate getAndReducedPredicate() { + return preFilters.stream().reduce((x, y) -> true, BiPredicate::and); + } + + public BiPredicate getOrReducedPredicate() { + return preFilters.stream().reduce((x, y) -> false, BiPredicate::or); + } + + public void clearFilters() { + preFilters.clear(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PrewarmRequest.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PrewarmRequest.java new file mode 100755 index 0000000000..0ba7be2c8d --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/PrewarmRequest.java @@ -0,0 +1,81 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Request that the database preload portions of the main vector table so that initial queries return faster from + * a server that has just been restarted. + * + */ +public class PrewarmRequest extends BSimQuery { + + public int mainIndexConfig; // For the main index -- 0=don't load 1=load into RAM 2=load into cache + public int secondaryIndexConfig; // For the secondary index -- 0=don't load 1=load into RAM 2=load into cache + public int vectorTableConfig; // For vectors -- 0=don't load 1=load into RAM 2=load into cache + public ResponsePrewarm prewarmresponse; + + public PrewarmRequest() { + super("prewarmrequest"); + // Set up default configuration + mainIndexConfig = 2; // Load into cache + secondaryIndexConfig = 1; // Load into any extra RAM + vectorTableConfig = 1; // Load into any extra RAM + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = prewarmresponse = new ResponsePrewarm(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + StringBuffer buffer = new StringBuffer(); + buffer.append('<').append(name).append(">\n"); + buffer.append("

"); + SpecXmlUtils.encodeSignedInteger(mainIndexConfig); + buffer.append("
\n"); + buffer.append(""); + SpecXmlUtils.encodeSignedInteger(secondaryIndexConfig); + buffer.append("\n"); + buffer.append(""); + SpecXmlUtils.encodeSignedInteger(vectorTableConfig); + buffer.append("
\n"); + buffer.append("\n"); + fwrite.write(buffer.toString()); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + parser.start(name); + parser.start("main"); + mainIndexConfig = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("secondary"); + secondaryIndexConfig = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("table"); + vectorTableConfig = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryChildren.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryChildren.java new file mode 100755 index 0000000000..1cd22dea8f --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryChildren.java @@ -0,0 +1,111 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Query based on a single executable and a specific list of functions names within the executable + * The response will be the corresponding FunctionDescription records and a record for each child + * of the specified functions + * + */ +public class QueryChildren extends BSimQuery { + + public String md5sum = null; + public String name_exec = null; + public String arch = null; + public String name_compiler = null; + public List functionKeys; + public ResponseChildren childrenresponse = null; + + public QueryChildren() { + super("querychildren"); + functionKeys = new ArrayList(); + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = childrenresponse = new ResponseChildren(this); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + if ((md5sum != null) && (md5sum.length() != 0)) { + fwrite.append(" ").append(md5sum).append("\n"); + } + else { + fwrite.append(" "); + if (name_exec != null) { + SpecXmlUtils.xmlEscapeWriter(fwrite, name_exec); + } + fwrite.append("\n"); + fwrite.append(" "); + if (arch != null) { + SpecXmlUtils.xmlEscapeWriter(fwrite, arch); + } + fwrite.append("\n"); + fwrite.append(" "); + if (name_compiler != null) { + SpecXmlUtils.xmlEscapeWriter(fwrite, name_compiler); + } + fwrite.append("\n"); + } + for (int i = 0; i < functionKeys.size(); ++i) { + functionKeys.get(i).saveXml(fwrite); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(name); + XmlElement el = parser.start(); + if (el.getName().equals("md5")) { + md5sum = parser.end().getText(); + name_exec = null; + arch = null; + name_compiler = null; + } + else { + md5sum = null; + name_exec = parser.end().getText(); + parser.start("arch"); + arch = parser.end().getText(); + parser.start("compiler"); + name_compiler = parser.end().getText(); + } + while (parser.peek().isStart()) { + FunctionEntry functionKey = FunctionEntry.restoreXml(parser); + functionKeys.add(functionKey); + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryCluster.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryCluster.java new file mode 100755 index 0000000000..433ebea979 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryCluster.java @@ -0,0 +1,86 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +public class QueryCluster extends BSimQuery { + + public final DescriptionManager manage; // Functions that should be queried as cluster roots + public ResponseCluster clusterresponse; + public double thresh; // Similarity limit of the cluster + public double signifthresh; // Significance limit of the cluster + public int vectormax; // Maximum number of vector results per function + + public QueryCluster() { + super("querycluster"); + manage = new DescriptionManager(); + thresh = 0.9; // Some reasonable defaults + signifthresh = 0.0; + vectormax = 50; + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = clusterresponse = new ResponseCluster(this); + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public QueryCluster getLocalStagingCopy() { + QueryCluster newc = new QueryCluster(); + newc.thresh = thresh; + newc.signifthresh = signifthresh; + newc.vectormax = vectormax; + return newc; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + manage.saveXml(fwrite); + fwrite.append("").append(Double.toString(thresh)).append("\n"); + fwrite.append("").append(Double.toString(signifthresh)).append("\n"); + fwrite.append("").append(SpecXmlUtils.encodeSignedInteger(vectormax)).append("\n"); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + parser.start(name); + manage.restoreXml(parser, vectorFactory); + parser.start("simthresh"); + thresh = Double.parseDouble(parser.end().getText()); + parser.start("signifthresh"); + signifthresh = Double.parseDouble(parser.end().getText()); + parser.start("max"); + vectormax = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryDelete.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryDelete.java new file mode 100755 index 0000000000..a3b2ddfaab --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryDelete.java @@ -0,0 +1,72 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.xml.XmlPullParser; + +/** + * Request that a specific list of executables be deleted from a BSim database + * + */ +public class QueryDelete extends BSimQuery { + + public List exelist; + public ResponseDelete respdelete; + + public QueryDelete() { + super("delete"); + exelist = new ArrayList(); + } + + public void addSpecifier(ExeSpecifier spec) { + exelist.add(spec); + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = respdelete = new ResponseDelete(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + Iterator iter = exelist.iterator(); + while (iter.hasNext()) { + iter.next().saveXml(fwrite); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + exelist = new ArrayList(); + parser.start(name); + while (parser.peek().isStart()) { + ExeSpecifier spec = new ExeSpecifier(); + exelist.add(spec); + spec.restoreXml(parser); + } + parser.end(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryExeCount.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryExeCount.java new file mode 100755 index 0000000000..6be759ba08 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryExeCount.java @@ -0,0 +1,89 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.xml.XmlPullParser; + +/** + * Query for counting the number of executable records in the database. + *

+ * This contains all the information required to get a list of all executables in the + * BSim database that meet a set of filter criteria. The results are stored in the + * {@link #exeresponse} object. + */ +public class QueryExeCount extends BSimQuery { + + public ResponseExe exeresponse = null; + public String filterMd5; + public String filterExeName; + public String filterArch; + public String filterCompilerName; + public boolean includeFakes; + + /** + * Query for count of all executables not including libraries + */ + public QueryExeCount() { + super("queryexecount"); + this.filterMd5 = null; + this.filterExeName = null; + this.filterArch = null; + this.filterCompilerName = null; + this.includeFakes = false; + } + + /** + * Constructor + * + * @param filterMd5 md5 filter + * @param filterExeName executable name filter + * @param filterArch architecture filter + * @param filterCompilerName compiler name filter + * @param includeFakes if true, include MD5s that start with bbbbbbbbaaaaaaa + */ + public QueryExeCount(String filterMd5, String filterExeName, String filterArch, + String filterCompilerName, boolean includeFakes) { + super("queryexecount"); + this.filterMd5 = filterMd5; + this.filterExeName = filterExeName; + this.filterArch = filterArch; + this.filterCompilerName = filterCompilerName; + this.includeFakes = includeFakes; + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = exeresponse = new ResponseExe(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + // no need to implement + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + // no need to implement + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryExeInfo.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryExeInfo.java new file mode 100755 index 0000000000..26eee2cd5c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryExeInfo.java @@ -0,0 +1,96 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.client.tables.ExeTable.ExeTableOrderColumn; +import ghidra.xml.XmlPullParser; + +/** + * Query of executable records + */ +public class QueryExeInfo extends BSimQuery { + + public ResponseExe exeresponse = null; + public int limit; + public String filterMd5; + public String filterExeName; + public String filterArch; + public String filterCompilerName; + public ExeTableOrderColumn sortColumn; + public boolean includeFakes; + public boolean fillinCategories; + + /** + * Default query for the first 20 executables in the database + */ + public QueryExeInfo() { + super("queryexeinfo"); + this.limit = 20; + this.filterMd5 = null; + this.filterExeName = null; + this.filterArch = null; + this.filterCompilerName = null; + this.sortColumn = ExeTableOrderColumn.MD5; + this.includeFakes = false; + this.fillinCategories = true; + } + + /** + * Constructor + * + * @param limit the max number of results to return + * @param filterMd5 md5 the md5 filter + * @param filterExeName the exe filter + * @param filterArch the architecture filter + * @param filterCompilerName the compiler name filter + * @param sortColumn the primary sort column name + * @param includeFakes if false, will exclude generated MD5s starting with "bbbbbbbbaaaaaaaa" + */ + public QueryExeInfo(int limit, String filterMd5, String filterExeName, String filterArch, + String filterCompilerName, ExeTableOrderColumn sortColumn, boolean includeFakes) { + super("queryexeinfo"); + this.limit = limit; + this.filterMd5 = filterMd5; + this.filterExeName = filterExeName; + this.filterArch = filterArch; + this.filterCompilerName = filterCompilerName; + this.sortColumn = sortColumn; + this.includeFakes = includeFakes; + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = exeresponse = new ResponseExe(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + // no need to implement + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + // no need to implement + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryInfo.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryInfo.java new file mode 100755 index 0000000000..692317d259 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryInfo.java @@ -0,0 +1,52 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.xml.XmlPullParser; + +/** + * Request the DatabaseInformation object for a specific BSim database. + * + */ +public class QueryInfo extends BSimQuery { + public ResponseInfo inforesponse; + + public QueryInfo() { + super("queryinfo"); + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = inforesponse = new ResponseInfo(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append("<").append(name).append("/>\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + // Nothing to do + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryName.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryName.java new file mode 100755 index 0000000000..7b8bdd5952 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryName.java @@ -0,0 +1,118 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Query for a single function in a single executable by giving either the md5 of the executable, or its name + * and version. Then give the name of the function. If the name of the function is empty, this query + * returns all functions in the executable + * + */ +public class QueryName extends BSimQuery { + + public ExeSpecifier spec; + public String funcname = ""; + public ResponseName nameresponse = null; + public int maxfunc; // Maximum function records to return + public boolean printselfsig; + public boolean printjustexe; + public boolean fillinSigs; + public boolean fillinCallgraph; + public boolean fillinCategories; + + public QueryName() { + super("queryname"); + spec = new ExeSpecifier(); + maxfunc = 1000; + printselfsig = false; + printjustexe = false; + fillinSigs = true; + fillinCallgraph = false; + fillinCategories = true; + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = nameresponse = new ResponseName(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + spec.saveXml(fwrite); + fwrite.append(""); + if (funcname != null) + SpecXmlUtils.xmlEscapeWriter(fwrite, funcname); + fwrite.append("\n"); + fwrite.append(""); + fwrite.append(Integer.toString(maxfunc)); + fwrite.append("\n"); + if (printselfsig) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + if (printjustexe) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + if (fillinSigs) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + if (fillinCallgraph) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + if (fillinCategories) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(name); + spec = new ExeSpecifier(); + spec.restoreXml(parser); + parser.start("funcname"); + funcname = parser.end().getText(); + parser.start("maxfunc"); + maxfunc = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("printselfsig"); + printselfsig = SpecXmlUtils.decodeBoolean(parser.end().getText()); + parser.start("printjustexe"); + printjustexe = SpecXmlUtils.decodeBoolean(parser.end().getText()); + parser.start("sigs"); + fillinSigs = SpecXmlUtils.decodeBoolean(parser.end().getText()); + parser.start("callgraph"); + fillinCallgraph = SpecXmlUtils.decodeBoolean(parser.end().getText()); + parser.start("categories"); + fillinCategories = SpecXmlUtils.decodeBoolean(parser.end().getText()); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryNearest.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryNearest.java new file mode 100755 index 0000000000..fda00105d6 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryNearest.java @@ -0,0 +1,146 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Query nearest matches within database to a set of functions + * + */ +public class QueryNearest extends BSimQuery { + + /** + * The default value for the similarity threshold. This + * threshold is for how similar the potential function is. This is a value from 0.0 to 1.0. + */ + public static final double DEFAULT_SIMILARITY_THRESHOLD = 0.7; + + /** + * The default value for the significance threshold. This + * threshold is for how significant the match is (for example, smaller function matches + * are less significant). Higher is more significant. There is no upper bound. + */ + public static final double DEFAULT_SIGNIFICANCE_THRESHOLD = 0.0; + + /** + * The default value for the maximum number of similar functions to return + * for a given input function + */ + public static final int DEFAULT_MAX_MATCHES = 100; + + public DescriptionManager manage; // Functions that should be queried + public ResponseNearest nearresponse; + public double thresh; // Similarity threshold for query + public double signifthresh; // Significance threshold for query + public int max; // Maximum number of results to return (per function) + public int vectormax; // Maximum number of unique vectors that can be returned + public boolean fillinCategories; // Query for categories of any returned executable + public BSimFilter bsimFilter; // Filters for the query + + public QueryNearest() { + super("querynearest"); + thresh = DEFAULT_SIMILARITY_THRESHOLD; + signifthresh = DEFAULT_SIGNIFICANCE_THRESHOLD; + max = DEFAULT_MAX_MATCHES; + vectormax = 0; // 0 indicates "no limit" + fillinCategories = true; + bsimFilter = null; + manage = new DescriptionManager(); + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = nearresponse = new ResponseNearest(this); + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public QueryNearest getLocalStagingCopy() { + QueryNearest newq = new QueryNearest(); + newq.thresh = thresh; + newq.signifthresh = signifthresh; + newq.max = max; + newq.vectormax = vectormax; + newq.fillinCategories = fillinCategories; + if (bsimFilter != null) + newq.bsimFilter = bsimFilter.clone(); + return newq; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + manage.saveXml(fwrite); + fwrite.append("").append(Double.toString(thresh)).append("\n"); + fwrite.append("").append(Double.toString(signifthresh)).append("\n"); + fwrite.append("").append(SpecXmlUtils.encodeSignedInteger(max)).append("\n"); + if (vectormax != 0) + fwrite.append("").append(SpecXmlUtils.encodeSignedInteger(vectormax)).append("\n"); + if (!fillinCategories) + fwrite.append("false\n"); + if (bsimFilter!=null) + bsimFilter.saveXml(fwrite); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + vectormax = 0; // Default + fillinCategories = true; // Default + parser.start(name); + manage.restoreXml(parser, vectorFactory); + parser.start("simthresh"); + thresh = Double.parseDouble(parser.end().getText()); + parser.start("signifthresh"); + signifthresh = Double.parseDouble(parser.end().getText()); + parser.start("max"); + max = SpecXmlUtils.decodeInt(parser.end().getText()); + while (parser.peek().isStart()) { + XmlElement el = parser.peek(); + if (el.getName().equals("vectormax")) { + parser.start(); + vectormax = SpecXmlUtils.decodeInt(parser.end().getText()); + } + else if (el.getName().equals("categories")) { + parser.start(); + fillinCategories = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + else if (el.getName().equals("exefilter")) { + bsimFilter = new BSimFilter(); + bsimFilter.restoreXml(parser); + } + else + throw new LSHException("Unknown tag: "+el.getName()); + + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryNearestVector.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryNearestVector.java new file mode 100755 index 0000000000..5e3c7520c3 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryNearestVector.java @@ -0,0 +1,101 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * For specific functions, query for the list of vectors that are similar to a functions vector, + * without recovering the descriptions of functions that instantiate these vectors. + * + */ +public class QueryNearestVector extends BSimQuery { + + public DescriptionManager manage; // Functions that should be queried + public ResponseNearestVector nearresponse; + public double thresh; // Similarity threshold for query + public double signifthresh; // Significance threshold for query + public int vectormax; // Maximum number of unique vectors returned + + public QueryNearestVector() { + super("querynearestvector"); + thresh = QueryNearest.DEFAULT_SIMILARITY_THRESHOLD; + signifthresh = QueryNearest.DEFAULT_SIGNIFICANCE_THRESHOLD; + vectormax = 0; // 0 indicates "no limit" + manage = new DescriptionManager(); + } + + + @Override + public void buildResponseTemplate() { + if (response == null) + response = nearresponse = new ResponseNearestVector(this); + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public QueryNearestVector getLocalStagingCopy() { + QueryNearestVector newq = new QueryNearestVector(); + newq.thresh = thresh; + newq.signifthresh = signifthresh; + newq.vectormax = vectormax; + return newq; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + manage.saveXml(fwrite); + fwrite.append("").append(Double.toString(thresh)).append("\n"); + fwrite.append("").append(Double.toString(signifthresh)).append("\n"); + if (vectormax != 0) + fwrite.append("").append(SpecXmlUtils.encodeSignedInteger(vectormax)).append("\n"); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + vectormax = 0; // Default + parser.start(name); + manage.restoreXml(parser, vectorFactory); + parser.start("simthresh"); + thresh = Double.parseDouble(parser.end().getText()); + parser.start("signifthresh"); + signifthresh = Double.parseDouble(parser.end().getText()); + while (parser.peek().isStart()) { + XmlElement el = parser.start(); + if (el.getName().equals("vectormax")) + vectormax = SpecXmlUtils.decodeInt(parser.end().getText()); + else + throw new LSHException("Unknown tag: "+el.getName()); + + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryOptionalExist.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryOptionalExist.java new file mode 100755 index 0000000000..0513086338 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryOptionalExist.java @@ -0,0 +1,99 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Query whether an optional table exists. If it doesn't exist it can be created. + * If it exists, it can be cleared + */ +public class QueryOptionalExist extends BSimQuery { + + public ResponseOptionalExist optionalresponse = null; + public String tableName; // Formal SQL name of the table + public int keyType; // type-code for the key column (from java.sql.Types) + public int valueType; // type-code for the value column + public boolean attemptCreation; // true if we should create the table if it doesn't exist + public boolean clearTable; // If true and table already exists, clear all rows of the table + + public QueryOptionalExist() { + super("queryoptionalexist"); + tableName = null; + keyType = -1; + valueType = -1; + attemptCreation = false; + clearTable = false; + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = optionalresponse = new ResponseOptionalExist(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + fwrite.append(""); + SpecXmlUtils.xmlEscapeWriter(fwrite, tableName); + fwrite.append("\n"); + fwrite.append(""); + fwrite.append(Integer.toString(keyType)); + fwrite.append("\n"); + fwrite.append(""); + fwrite.append(Integer.toString(valueType)); + fwrite.append("\n"); + if (attemptCreation) { + fwrite.append("true\n"); + } + else { + fwrite.append("false\n"); + } + if (clearTable) { + fwrite.append("true\n"); + } + else { + fwrite.append("false\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(name); + parser.start("tablename"); + tableName = parser.end().getText(); + parser.start("keytype"); + keyType = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("valuetype"); + valueType = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("create"); + attemptCreation = SpecXmlUtils.decodeBoolean(parser.end().getText()); + parser.start("clear"); + clearTable = SpecXmlUtils.decodeBoolean(parser.end().getText()); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryOptionalValues.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryOptionalValues.java new file mode 100755 index 0000000000..4b63a37395 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryOptionalValues.java @@ -0,0 +1,98 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Query for values from an optional table, given a set of keys + */ +public class QueryOptionalValues extends BSimQuery { + + public ResponseOptionalValues optionalresponse = null; + public Object[] keys; // Keys to query + public String tableName; // Name of the optional table + public int keyType; // Type of the key, as per java.sql.Types + public int valueType; // Type of the value + + public QueryOptionalValues() { + super("queryoptionalvalues"); + tableName = null; + keys = null; + keyType = -1; + valueType = -1; + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = optionalresponse = new ResponseOptionalValues(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + fwrite.append(""); + SpecXmlUtils.xmlEscapeWriter(fwrite, tableName); + fwrite.append("\n"); + fwrite.append(""); + fwrite.append(Integer.toString(keyType)); + fwrite.append("\n"); + fwrite.append(""); + fwrite.append(Integer.toString(valueType)); + fwrite.append("\n"); + for (Object key : keys) { + fwrite.append(""); + SpecXmlUtils.xmlEscapeWriter(fwrite, key.toString()); + fwrite.append("\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + keys = null; + List resultKeys = new ArrayList(); + parser.start(name); + parser.start("tablename"); + tableName = parser.end().getText(); + parser.start("keytype"); + keyType = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("valuetype"); + valueType = SpecXmlUtils.decodeInt(parser.end().getText()); + while (parser.peek().isStart()) { + parser.start(); + String key = parser.end().getText(); + resultKeys.add(key); + } + parser.end(); + if (!resultKeys.isEmpty()) { + keys = new Object[resultKeys.size()]; + resultKeys.toArray(keys); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryPair.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryPair.java new file mode 100755 index 0000000000..1a28d2075c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryPair.java @@ -0,0 +1,72 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * A list of descriptor pairs to be sent to the server. + * Each pair describes a pair of functions in the database whose vectors are to be compared + * + */ +public class QueryPair extends BSimQuery { + + public List pairs; + public ResponsePair pairResponse; + + public QueryPair() { + super("querypair"); + pairs = new ArrayList(); + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = pairResponse = new ResponsePair(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + for (PairInput pairInput : pairs) { + pairInput.saveXml(fwrite); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + XmlElement startEl = parser.start(name); + for (;;) { + XmlElement note = parser.peek(); + if (!note.isStart()) + break; + PairInput pairInput = new PairInput(); + pairInput.restoreXml(parser); + pairs.add(pairInput); + } + parser.end(startEl); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryResponseRecord.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryResponseRecord.java new file mode 100755 index 0000000000..66a2b0c3b2 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryResponseRecord.java @@ -0,0 +1,64 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.xml.XmlPullParser; + +// A database query response record that can be serialized + +public abstract class QueryResponseRecord { + + protected final String name; + + protected QueryResponseRecord(String name) { + this.name = name; + } + + public String getName() { return name; } + + public abstract void saveXml(Writer fwrite) throws IOException; + + public abstract void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException; + + public DescriptionManager getDescriptionManager() { return null; } + + /** + * @return a partial clone of this query suitable for holding local stages of the query via StagingManager + */ + public QueryResponseRecord getLocalStagingCopy() { return null; } + + /** + * Combine partial results from subresponse into this global response + * @param subresponse the partial response to merge into this + * @throws LSHException for errors performing the merge + */ + public void mergeResults(QueryResponseRecord subresponse) throws LSHException { + // Must subclasses don't need to do anything + } + + /** + * Perform any preferred sorting on the result of a query + */ + public void sort() { + // Must subclasses don't need to do this + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryUpdate.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryUpdate.java new file mode 100755 index 0000000000..587efe8361 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryUpdate.java @@ -0,0 +1,75 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.xml.XmlPullParser; + +/** + * Request to update the metadata fields of various ExecutableRecords and FunctionDescriptions within a BSim database. + * This allows quick updates of metadata fields like executable names, function names, and other descriptive metadata fields, + * without affecting the main index. ExecutableRecord descriptions will be replaced based on the md5 of the executable, + * and FunctionDescriptions are replaced based on their address within an identified executable. + * within an executable. + * + */ +public class QueryUpdate extends BSimQuery { + + public DescriptionManager manage; // contains the list of ExecutableRecords and FunctionDescriptions to update + public ResponseUpdate updateresponse; + + public QueryUpdate() { + super("update"); + manage = new DescriptionManager(); + } + + @Override + public void buildResponseTemplate() { + if (response == null) + response = updateresponse = new ResponseUpdate(this); + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public QueryUpdate getLocalStagingCopy() { + QueryUpdate newq = new QueryUpdate(); + return newq; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + manage.saveXml(fwrite); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + parser.start(name); + manage.restoreXml(parser, vectorFactory); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryVectorId.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryVectorId.java new file mode 100755 index 0000000000..ac9c55b1a0 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryVectorId.java @@ -0,0 +1,72 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Request vectors from the database by the their ids. Allows users to retrieve raw feature + * vectors without going through functions (FunctionDescription and DescriptionManager) + */ +public class QueryVectorId extends BSimQuery { + + public List vectorIds; // The list of ids to query for + public ResponseVectorId vectorIdResponse; + + public QueryVectorId() { + super("queryvectorid"); + vectorIds = new ArrayList(); + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = vectorIdResponse = new ResponseVectorId(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + for (Long id : vectorIds) { + fwrite.append(" 0x"); + fwrite.append(Long.toHexString(id.longValue())); + fwrite.append("\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(); + while (parser.peek().isStart()) { + parser.start(); + long val = SpecXmlUtils.decodeLong(parser.end().getText()); + vectorIds.add(val); + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryVectorMatch.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryVectorMatch.java new file mode 100755 index 0000000000..367405c36c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/QueryVectorMatch.java @@ -0,0 +1,107 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Request all functions described by a particular feature vector. Vectors are specified + * by a known id, and multiple vectors can be specified at once. + */ +public class QueryVectorMatch extends BSimQuery { + + // Default maximum number of functions to return that match a single vector id + public static final int DEFAULT_MAX_FUNCTIONS = 200; + public ResponseVectorMatch matchresponse; + public int max; // Maximum number of results to return (per vector id) + public boolean fillinCategories; // Query for categories of any returned executable + public BSimFilter bsimFilter; // Filters for the query + public List vectorIds; // List of vector ids to query for + + public QueryVectorMatch() { + super("queryvectormatch"); + max = DEFAULT_MAX_FUNCTIONS; + fillinCategories = true; + bsimFilter = null; + vectorIds = new ArrayList(); + } + + @Override + public void buildResponseTemplate() { + if (response == null) { + response = matchresponse = new ResponseVectorMatch(); + } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + fwrite.append("").append(SpecXmlUtils.encodeSignedInteger(max)).append("\n"); + if (!fillinCategories) { + fwrite.append("false\n"); + } + if (bsimFilter != null) { + bsimFilter.saveXml(fwrite); + } + for (Long id : vectorIds) { + fwrite.append("0x") + .append(SpecXmlUtils.encodeUnsignedInteger(id)) + .append( + "\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + fillinCategories = true; // Default + parser.start(name); + parser.start("max"); + max = SpecXmlUtils.decodeInt(parser.end().getText()); + while (parser.peek().isStart()) { + XmlElement el = parser.peek(); + if (el.getName().equals("categories")) { + parser.start(); + fillinCategories = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + else if (el.getName().equals("exefilter")) { + bsimFilter = new BSimFilter(); + bsimFilter.restoreXml(parser); + } + else if (el.getName().equals("id")) { + parser.start(); + long val = SpecXmlUtils.decodeLong(parser.end().getText()); + vectorIds.add(val); + } + else { + throw new LSHException("Unknown tag: " + el.getName()); + } + + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseAdjustIndex.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseAdjustIndex.java new file mode 100755 index 0000000000..07a77b2fa9 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseAdjustIndex.java @@ -0,0 +1,59 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Response to an AdjustVectorIndex request, returning a boolean value of either success or failure of the request + * + */ +public class ResponseAdjustIndex extends QueryResponseRecord { + + public boolean success; + public boolean operationSupported; // true if the back-end supports this operation + + public ResponseAdjustIndex() { + super("responseadjust"); + success = false; + operationSupported = true; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(" success=\""); + fwrite.append(SpecXmlUtils.encodeBoolean(success)); + fwrite.append("\" support=\""); + fwrite.append(SpecXmlUtils.encodeBoolean(operationSupported)); + fwrite.append("\"/>\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + XmlElement el = parser.start(name); + success = SpecXmlUtils.decodeBoolean(el.getAttribute("success")); + operationSupported = SpecXmlUtils.decodeBoolean(el.getAttribute("support")); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseChildren.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseChildren.java new file mode 100755 index 0000000000..0a34e0cc95 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseChildren.java @@ -0,0 +1,77 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; +import ghidra.xml.XmlPullParser; + +/** + * Response to a QueryChildren request to a BSim database. A full FunctionDescription is returned for + * every name in the original request and their children (1-level). The FunctionDescriptions corresponding + * to the original list of function names are also collected in the -correspond- array. + * + */ +public class ResponseChildren extends QueryResponseRecord { + + public DescriptionManager manage; // A description of the originally requested functions and their children + public List correspond; // The list of originally requested FunctionDescriptions + public QueryChildren qchild; // The original query for which this is a response + + public ResponseChildren(QueryChildren qc) { + super("responsechildren"); + manage = new DescriptionManager(); + correspond = new ArrayList(); + qchild = qc; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + manage.saveXml(fwrite); + if (!correspond.isEmpty()) { + fwrite.append(""); + fwrite.append(correspond.get(0).getExecutableRecord().getMd5()); + fwrite.append("\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(name); + manage.restoreXml(parser, vectorFactory); + if (!parser.peek().isStart()) { + return; + } + parser.start("md5"); + String md5string = parser.end().getText(); + + ExecutableRecord exe = manage.findExecutable(md5string); + for (FunctionEntry entry : qchild.functionKeys) { + correspond.add(manage.findFunction(entry.funcName, entry.address, exe)); + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseCluster.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseCluster.java new file mode 100755 index 0000000000..793b50ee34 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseCluster.java @@ -0,0 +1,62 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.xml.XmlPullParser; + +public class ResponseCluster extends QueryResponseRecord { + + public List notes; + public QueryCluster query; + + public ResponseCluster(QueryCluster q) { + super("responsecluster"); + notes = new ArrayList(); + query = q; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + query.manage.populateExecutableXref(); // Make cross-references are pregenerated + fwrite.append('<').append(name).append(">\n"); + Iterator iter = notes.iterator(); + while (iter.hasNext()) { + iter.next().saveXml(fwrite); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + Map exeMap = query.manage.generateExecutableXrefMap(); + parser.start(name); + while (parser.peek().isStart()) { + ClusterNote newnote = new ClusterNote(); + newnote.restoreXml(parser, query.manage, exeMap); + notes.add(newnote); + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseDelete.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseDelete.java new file mode 100755 index 0000000000..62be8c982c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseDelete.java @@ -0,0 +1,109 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Response to a QueryDelete request containing a listing of the md5's of successfully deleted executables and + * a count of their functions. If a requested executable could not be deleted for some reason it is listed in + * a separate -missedlist- + * + */ +public class ResponseDelete extends QueryResponseRecord { + + public static class DeleteResult { + public String md5; // Md5 of executable successfully deleted + public String name; // name of deleted executable + public int funccount; // Number of (now deleted) function records associated with executable + + public void saveXml(Writer fwrite) throws IOException { + fwrite.append("\n"); + fwrite.append(" ").append(md5).append("\n"); + fwrite.append(" "); + SpecXmlUtils.xmlEscapeWriter(fwrite, name); + fwrite.append("\n"); + fwrite.append(" ") + .append(SpecXmlUtils.encodeSignedInteger(funccount)) + .append("\n"); + fwrite.append("\n"); + } + + public void restoreXml(XmlPullParser parser) { + parser.start("delrec"); + parser.start("md5"); + md5 = parser.end().getText(); + parser.start("name"); + name = parser.end().getText(); + parser.start("count"); + funccount = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.end(); + } + } + + public List reslist; // List of executables successfully deleted + public List missedlist; // List of executables that could not be deleted + + public ResponseDelete() { + super("responsedelete"); + reslist = new ArrayList(); + missedlist = new ArrayList(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + Iterator iter = reslist.iterator(); + while (iter.hasNext()) { + iter.next().saveXml(fwrite); + } + Iterator miter = missedlist.iterator(); + while (miter.hasNext()) { + miter.next().saveXml(fwrite); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + reslist = new ArrayList(); + missedlist = new ArrayList(); + parser.start(name); + while (parser.peek().isStart()) { + XmlElement el = parser.peek(); + if (el.getName().equals("delrec")) { + DeleteResult res = new DeleteResult(); + res.restoreXml(parser); + reslist.add(res); + } + else { + ExeSpecifier spec = new ExeSpecifier(); + spec.restoreXml(parser); + } + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseError.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseError.java new file mode 100755 index 0000000000..795e63e53c --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseError.java @@ -0,0 +1,47 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +public class ResponseError extends QueryResponseRecord { + public String errorMessage; + + public ResponseError() { + super("error"); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + SpecXmlUtils.xmlEscapeWriter(fwrite,errorMessage); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + XmlElement el = parser.start(name); + errorMessage = parser.end(el).getText(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseExe.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseExe.java new file mode 100755 index 0000000000..9106c5c59a --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseExe.java @@ -0,0 +1,64 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.ingest.BulkSignatures; +import ghidra.xml.XmlPullParser; + +/** + * Response to a request for executables from a {@link BulkSignatures} call. + * + */ +public class ResponseExe extends QueryResponseRecord { + + public List records; + public DescriptionManager manage; + public int recordCount = 0; + + /** + * Constructor. + */ + public ResponseExe() { + super("responsexe"); + manage = new DescriptionManager(); + records = new ArrayList<>(); + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + // no need to implement + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + // no need to implement + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseInfo.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseInfo.java new file mode 100755 index 0000000000..e04aa2e273 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseInfo.java @@ -0,0 +1,49 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.xml.XmlPullParser; + +public class ResponseInfo extends QueryResponseRecord { + public DatabaseInformation info; + + public ResponseInfo() { + super("responseinfo"); + info = null; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + info.saveXml(fwrite); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + info = new DatabaseInformation(); + parser.start(name); + info.restoreXml(parser); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseInsert.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseInsert.java new file mode 100755 index 0000000000..78c50f8d4b --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseInsert.java @@ -0,0 +1,65 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * A simple response to an InsertRequest to a BSim database. + * This object provides separate counts of executables successfully inserted and functions successfully inserted. + * + */ +public class ResponseInsert extends QueryResponseRecord { + + public int numexe; // Number of executables inserted + public int numfunc; // NUmber of functions inserted + + public ResponseInsert() { + super("responseinsert"); + } + + @Override + public void mergeResults(QueryResponseRecord subresponse) { + ResponseInsert subinsert = (ResponseInsert)subresponse; + numexe += subinsert.numexe; + numfunc += subinsert.numfunc; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + fwrite.append(" ").append(SpecXmlUtils.encodeSignedInteger(numexe)).append("\n"); + fwrite.append(" ").append(SpecXmlUtils.encodeSignedInteger(numfunc)).append("\n"); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + parser.start(name); + parser.start("numexe"); + numexe = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("numfunc"); + numfunc = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseName.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseName.java new file mode 100755 index 0000000000..67f435dd5b --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseName.java @@ -0,0 +1,131 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.*; +import java.util.Iterator; +import java.util.TreeSet; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Response to a request for specific executables and functions given by name. + * Full ExecutableRecords and FunctionDescriptions are instantiated in this object's DescriptionManager + * + */ +public class ResponseName extends QueryResponseRecord { + + public final DescriptionManager manage; // Set of functions and executables matching the name request + public boolean uniqueexecutable; // True if query specified a unique executable + public boolean printselfsig; + public boolean printjustexe; + + public ResponseName() { + super("responsename"); + manage = new DescriptionManager(); + uniqueexecutable = false; + printselfsig = false; + printjustexe = false; + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + if (uniqueexecutable) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + if (printselfsig) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + if (printjustexe) + fwrite.append("true\n"); + else + fwrite.append("false\n"); + manage.saveXml(fwrite); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + uniqueexecutable = false; + printselfsig = false; + printjustexe = false; + parser.start(name); + if (parser.peek().getName().equals("uniqueexe")) { + parser.start(); + uniqueexecutable = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + if (parser.peek().getName().equals("printselfsig")) { + parser.start(); + printselfsig = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + if (parser.peek().getName().equals("printjustexe")) { + parser.start(); + printjustexe = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + manage.restoreXml(parser, vectorFactory); + parser.end(); + } + + public void printRaw(PrintStream stream, LSHVectorFactory vectorFactory, int format) { + if (!uniqueexecutable) { + stream.println("Unable to resolve unique executable"); + } + if ((!uniqueexecutable) || printjustexe) { + TreeSet exeset = manage.getExecutableRecordSet(); + Iterator iter = exeset.iterator(); + while (iter.hasNext()) { + String line = iter.next().printRaw(); + stream.println(line); + } + return; + } + ExecutableRecord lastexe = null; + Iterator iter = manage.listAllFunctions(); + while (iter.hasNext()) { + FunctionDescription funcrec = iter.next(); + if (lastexe != funcrec.getExecutableRecord()) { + lastexe = funcrec.getExecutableRecord(); + String line = lastexe.printRaw(); + stream.println(line); + } + stream.print(" "); + if (printselfsig) { + double val = 0.0; + SignatureRecord srec = funcrec.getSignatureRecord(); + if (srec != null) + val = vectorFactory.getSelfSignificance(srec.getLSHVector()); + stream.print(val); + stream.print(' '); + } + String line = funcrec.printRaw(); + stream.println(line); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseNearest.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseNearest.java new file mode 100755 index 0000000000..6f9e1e5104 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseNearest.java @@ -0,0 +1,121 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Response to a QueryNearest request. A full description in terms of ExecutableRecords and FunctionDescriptions + * is returned. The linked list of SimilarityResults explicitly describes the similarities between the functions + * in the original request and the new functions being returned. A SimilarityResult cross-references + * FunctionDescription objects between the request DescriptionManager and this response object's DescriptionManager + * + */ +public class ResponseNearest extends QueryResponseRecord { + + public int totalfunc; // Total functions queried + public int totalmatch; // Total number of functions matched + public int uniquematch; // Total number of functions matched uniquely + public final DescriptionManager manage; // The collection of matching functions + public List result; // Description of similarities + public QueryNearest qnear; // Original query + + public ResponseNearest(QueryNearest q) { + super("responsenearest"); + manage = new DescriptionManager(); + result = new ArrayList(); + totalfunc = 0; + totalmatch = 0; + uniquematch = 0; + qnear = q; + } + + @Override + public void sort() { + for (SimilarityResult sim : result) + sim.sortNotes(); + } + + @Override + public void mergeResults(QueryResponseRecord subresponse) throws LSHException { + ResponseNearest subnearest = (ResponseNearest) subresponse; + if (totalfunc == 0) + manage.transferSettings(subnearest.manage); // Transfer settings first time through + totalfunc += subnearest.totalfunc; + totalmatch += subnearest.totalmatch; + uniquematch += subnearest.uniquematch; + result.addAll(subnearest.result); + + // Substitute the above result.addAll line with the commented code below for RegressionTestQuery test of staging +// for(SimilarityResult simres : subnearest.result) { +// SimilarityResult newsimres = new SimilarityResult(); +// newsimres.setTransfer(simres, qnear.manage, manage, false); +// result.add(newsimres); +// } + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + manage.populateExecutableXref(); + qnear.manage.populateExecutableXref(); + fwrite.append('<').append(name).append(">\n"); + fwrite.append(" ") + .append(SpecXmlUtils.encodeSignedInteger(totalfunc)) + .append("\n"); + fwrite.append(" ") + .append(SpecXmlUtils.encodeSignedInteger(totalmatch)) + .append("\n"); + fwrite.append(" ") + .append(SpecXmlUtils.encodeSignedInteger(uniquematch)) + .append("\n"); + manage.saveXml(fwrite); + Iterator iter = result.iterator(); + while (iter.hasNext()) { + iter.next().saveXml(fwrite); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(name); + parser.start("tfunc"); + totalfunc = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("tmatch"); + totalmatch = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("umatch"); + uniquematch = SpecXmlUtils.decodeInt(parser.end().getText()); + manage.restoreXml(parser, vectorFactory); + Map qMap = qnear.manage.generateExecutableXrefMap(); + Map rMap = manage.generateExecutableXrefMap(); + while (parser.peek().isStart()) { + SimilarityResult res = new SimilarityResult(); + res.restoreXml(parser, qnear.manage, manage, qMap, rMap); + result.add(res); + } + parser.end(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseNearestVector.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseNearestVector.java new file mode 100755 index 0000000000..99ce413814 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseNearestVector.java @@ -0,0 +1,103 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Response to a QueryNearestVector request. It provides basic stats on the number of matching vectors and functions. + * Only a list of the matching vectors is returned, not the detailed FunctionDescription records of matches. + * Results are returned as SimilarityVectorResult objects, which cross-reference the original function queried and + * any similar vectors. + * + */ +public class ResponseNearestVector extends QueryResponseRecord { + public int totalvec; // Total vectors queried + public int totalmatch; // Total functions matched + public int uniquematch; // Total vectors with a unique function match + public List result; + public QueryNearestVector qnear; + + public ResponseNearestVector(QueryNearestVector q) { + super("responsenearestvec"); + result = new ArrayList(); + qnear = q; + } + + @Override + public void sort() { + for (SimilarityVectorResult res : result) { + res.sortNotes(); + } + } + + @Override + public void mergeResults(QueryResponseRecord subresponse) { + ResponseNearestVector subnearest = (ResponseNearestVector) subresponse; + totalvec += subnearest.totalvec; + totalmatch += subnearest.totalmatch; + uniquematch += subnearest.uniquematch; + result.addAll(subnearest.result); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + qnear.manage.populateExecutableXref(); + fwrite.append('<').append(name).append(">\n"); + fwrite.append(" ") + .append(SpecXmlUtils.encodeSignedInteger(totalvec)) + .append("\n"); + fwrite.append(" ") + .append(SpecXmlUtils.encodeSignedInteger(totalmatch)) + .append("\n"); + fwrite.append(" ") + .append(SpecXmlUtils.encodeSignedInteger(uniquematch)) + .append("\n"); + Iterator iter = result.iterator(); + while (iter.hasNext()) { + iter.next().saveXml(fwrite); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(name); + parser.start("tvec"); + totalvec = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("tmatch"); + totalmatch = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("umatch"); + uniquematch = SpecXmlUtils.decodeInt(parser.end().getText()); + Map exeMap = qnear.manage.generateExecutableXrefMap(); + while (parser.peek().isStart()) { + SimilarityVectorResult res = new SimilarityVectorResult(); + res.restoreXml(parser, vectorFactory, qnear.manage, exeMap); + result.add(res); + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseOptionalExist.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseOptionalExist.java new file mode 100755 index 0000000000..400146069b --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseOptionalExist.java @@ -0,0 +1,75 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +/** + * Response to a QueryOptionalExist, reporting whether an optional table exists + */ +public class ResponseOptionalExist extends QueryResponseRecord { + + public boolean tableExists; // true if the queried table exists + public boolean wasCreated; // true if this query caused creation of table + + public ResponseOptionalExist() { + super("responseoptionalexist"); + tableExists = false; + wasCreated = false; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + if (tableExists) { + fwrite.append("true\n"); + } + else { + fwrite.append("false\n"); + } + if (wasCreated) { + fwrite.append("true\n"); + } + else { + fwrite.append("false\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + tableExists = false; + wasCreated = false; + parser.start(name); + if (parser.peek().getName().equals("exists")) { + parser.start(); + tableExists = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + if (parser.peek().getName().equals("created")) { + parser.start(); + tableExists = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseOptionalValues.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseOptionalValues.java new file mode 100755 index 0000000000..69d84a9857 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseOptionalValues.java @@ -0,0 +1,80 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlPullParser; + +public class ResponseOptionalValues extends QueryResponseRecord { + + // FIXME: XML serialization assumes String-based resultArray which is incorrect + + public Object[] resultArray; // Array of values corresponding to queried keys + public boolean tableExists; // false if the query failed because the table doesn't exist + + public ResponseOptionalValues() { + super("responseoptionalvalues"); + resultArray = null; + tableExists = true; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + if (!tableExists) { + fwrite.append("false\n"); + } + if (resultArray != null) { + for (Object value : resultArray) { + fwrite.append(""); + SpecXmlUtils.xmlEscapeWriter(fwrite, value.toString()); + fwrite.append("\n"); + } + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + tableExists = true; + resultArray = null; + List resValues = new ArrayList(); + parser.start(name); + if (parser.peek().getName().equals("exists")) { + parser.start("exists"); + tableExists = SpecXmlUtils.decodeBoolean(parser.end().getText()); + } + while (parser.peek().isStart()) { + parser.start(); + String value = parser.end().getText(); + resValues.add(value); + } + parser.end(); + if (!resValues.isEmpty()) { + resultArray = new Object[resValues.size()]; + resValues.toArray(resultArray); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePair.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePair.java new file mode 100755 index 0000000000..3e84dc3786 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePair.java @@ -0,0 +1,158 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * A list of records (PairNote) each describing the comparison of a pair of functions on the server + * This response also includes various statistics (counts and averages) on the results + * + */ +public class ResponsePair extends QueryResponseRecord { + + public double averageSim; // Average similarity of pairs + public double simStdDev; // Similarity standard deviation + public double averageSig; // Average significance of pairs + public double sigStdDev; // Significance standard deviation + public double scale; // Weight table scale used in comparison + public int pairCount; // Valid pairs + public int missedExe; // Number of executables that could not be resolved + public int missedFunc; // Number of functions that could not be resolved + public int missedVector; // Number of functions without a vector to compare + public List notes; + + public static class Accumulator { + public double sumSim = 0.0; + public double sumSimSquare = 0.0; + public double sumSig = 0.0; + public double sumSigSquare = 0.0; + public int missedExe = 0; + public int missedFunc = 0; + public int missedVector = 0; + public int pairCount = 0; + + /** + * Accumulate from already summarized statistics in a ResponsePair + * This method can be called multiple times to aggregate responses from multiple ResponsePairs + * @param responsePair to be merged + */ + public void merge(ResponsePair responsePair) { + pairCount += responsePair.pairCount; + missedExe += responsePair.missedExe; + missedFunc += responsePair.missedFunc; + missedVector += responsePair.missedVector; + sumSim += responsePair.averageSim * responsePair.pairCount; + sumSig += responsePair.averageSig * responsePair.pairCount; + double aveSimSquare = responsePair.simStdDev * responsePair.simStdDev + + responsePair.averageSim * responsePair.averageSim; + sumSimSquare += aveSimSquare * responsePair.pairCount; + double aveSigSquare = responsePair.sigStdDev * responsePair.sigStdDev + + responsePair.averageSig * responsePair.averageSig; + sumSigSquare += aveSigSquare * responsePair.pairCount; + } + } + + public ResponsePair() { + super("responsepair"); + notes = new ArrayList(); + } + + public void fillOutStatistics(Accumulator accumulator) { + pairCount = accumulator.pairCount; + averageSim = accumulator.sumSim / pairCount; + averageSig = accumulator.sumSig / pairCount; + simStdDev = Math.sqrt(accumulator.sumSimSquare / pairCount - averageSim * averageSim); + sigStdDev = Math.sqrt(accumulator.sumSigSquare / pairCount - averageSig * averageSig); + missedExe = accumulator.missedExe; + missedFunc = accumulator.missedFunc; + missedVector = accumulator.missedVector; + } + + public void saveXmlTail(Writer fwrite) throws IOException { + fwrite.append(" ").append(Double.toString(averageSim)).append("\n"); + fwrite.append(" ").append(Double.toString(simStdDev)).append("\n"); + fwrite.append(" ").append(Double.toString(averageSig)).append("\n"); + fwrite.append(" ").append(Double.toString(sigStdDev)).append("\n"); + fwrite.append(" ").append(Double.toString(scale)).append("\n"); + fwrite.append(" ").append(Integer.toString(pairCount)).append("\n"); + fwrite.append(" ").append(Integer.toString(missedExe)).append("\n"); + fwrite.append(" ") + .append(Integer.toString(missedFunc)) + .append("\n"); + fwrite.append(" ") + .append(Integer.toString(missedVector)) + .append( + "\n"); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + for (PairNote note : notes) { + note.saveXml(fwrite); + } + saveXmlTail(fwrite); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + XmlElement startEl = parser.start(name); + for (;;) { + XmlElement note = parser.peek(); + if (!note.isStart()) { + break; + } + if (!note.getName().equals("note")) { + break; + } + PairNote pairNote = new PairNote(); + pairNote.restoreXml(parser); + notes.add(pairNote); + } + parser.start("avesim"); + averageSim = Double.parseDouble(parser.end().getText()); + parser.start("simstddev"); + simStdDev = Double.parseDouble(parser.end().getText()); + parser.start("avesig"); + averageSig = Double.parseDouble(parser.end().getText()); + parser.start("sigstddev"); + sigStdDev = Double.parseDouble(parser.end().getText()); + parser.start("scale"); + scale = Double.parseDouble(parser.end().getText()); + parser.start("paircount"); + pairCount = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("missedexe"); + missedExe = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("missedfunc"); + missedFunc = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.start("missedvector"); + missedVector = SpecXmlUtils.decodeInt(parser.end().getText()); + parser.end(startEl); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePassword.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePassword.java new file mode 100755 index 0000000000..824cfe0304 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePassword.java @@ -0,0 +1,64 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Response of server indicating whether a password change request ({@link PasswordChange}) succeeded + */ +public class ResponsePassword extends QueryResponseRecord { + + public boolean changeSuccessful; // true if the password change was successful + public String errorMessage; // Error message if change was not successful + + public ResponsePassword() { + super("responsepassword"); + changeSuccessful = false; + errorMessage = null; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name); + fwrite.append(" success=\""); + SpecXmlUtils.encodeBoolean(changeSuccessful); + fwrite.append("\">"); + if (errorMessage != null) { + SpecXmlUtils.xmlEscapeWriter(fwrite, errorMessage); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + XmlElement el = parser.start(name); + changeSuccessful = SpecXmlUtils.decodeBoolean(el.getAttribute("success")); + errorMessage = parser.end().getText(); + if (errorMessage != null && errorMessage.length() == 0) { + errorMessage = null; + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePrewarm.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePrewarm.java new file mode 100755 index 0000000000..2719cf98ad --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponsePrewarm.java @@ -0,0 +1,58 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Response to a PrewarmRequest indicating that number of database blocks that were preloaded + * + */ +public class ResponsePrewarm extends QueryResponseRecord { + + public int blockCount; // Number of blocks in main index that were read + public boolean operationSupported; // true if the back-end supports this operation + + public ResponsePrewarm() { + super("responseprewarm"); + blockCount = -1; + operationSupported = true; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name); + fwrite.append(" support=\"").append(SpecXmlUtils.encodeBoolean(operationSupported)); + fwrite.append("\">\n"); + Integer.toString(blockCount); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) throws LSHException { + XmlElement el = parser.start(name); + operationSupported = SpecXmlUtils.decodeBoolean(el.getAttribute("support")); + blockCount = SpecXmlUtils.decodeInt(parser.end().getText()); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseUpdate.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseUpdate.java new file mode 100755 index 0000000000..cc1d66a402 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseUpdate.java @@ -0,0 +1,104 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * Response to a QueryUpdate request to a BSim database. Simple counts of successful updates are given. + * References to any original ExecutableRecord or FunctionDescription objects that could not be updated + * are also returned. + * + */ +public class ResponseUpdate extends QueryResponseRecord { + + public List badexe; + public List badfunc; + public int exeupdate; // Number of executable records updated + public int funcupdate; // Number of function records updated + public QueryUpdate qupdate; // Original query + + public ResponseUpdate(QueryUpdate q) { + super("responseupdate"); + badexe = new ArrayList(); + badfunc = new ArrayList(); + exeupdate = 0; + funcupdate = 0; + qupdate = q; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + qupdate.manage.populateExecutableXref(); // Make sure cross-references are pregenerated + fwrite.append('<').append(name).append(">\n"); + Iterator iter2 = badexe.iterator(); + while (iter2.hasNext()) { + ExecutableRecord exe = iter2.next(); + fwrite.append("\n"); + } + Iterator iter = badfunc.iterator(); + while (iter.hasNext()) { + FunctionDescription func = iter.next(); + fwrite.append("\n"); + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + + DescriptionManager manage = qupdate.manage; + Map exeMap = manage.generateExecutableXrefMap(); + parser.start(); + while (parser.peek().isStart()) { + XmlElement el = parser.start(); + if (el.getName().equals("badexe")) { + int id = SpecXmlUtils.decodeInt(el.getAttribute("id")); + ExecutableRecord exe = exeMap.get(id); + badexe.add(exe); + } + else if (el.getName().equals("badfunc")) { + int id = SpecXmlUtils.decodeInt(el.getAttribute("id")); + long address = SpecXmlUtils.decodeLong(el.getAttribute("addr")); + ExecutableRecord exe = exeMap.get(id); + FunctionDescription func = + manage.findFunction(el.getAttribute("name"), address, exe); + badfunc.add(func); + } + parser.end(); + } + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseVectorId.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseVectorId.java new file mode 100755 index 0000000000..dc7828e7e8 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseVectorId.java @@ -0,0 +1,75 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.ArrayList; +import java.util.List; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.VectorResult; +import ghidra.xml.XmlPullParser; + +/** + * Response to a QueryVectorId request to a BSim database. For each id in the + * request, return a VectorResult, which contains the corresponding full vector, + * or return null + * + */ +public class ResponseVectorId extends QueryResponseRecord { + + public List vectorResults; // List of result objects (or null) one per requested id + + public ResponseVectorId() { + super("responsevectorid"); + vectorResults = new ArrayList(); + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + for (VectorResult vecResult : vectorResults) { + if (vecResult == null) { + fwrite.append(" \n"); + } + else { + vecResult.saveXml(fwrite); + } + } + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(); + while (parser.peek().isStart()) { + if (parser.peek().getName().equals("null")) { + parser.discardSubTree(); + vectorResults.add(null); + } + else { + VectorResult vecResult = new VectorResult(); + vecResult.restoreXml(parser, vectorFactory); + vectorResults.add(vecResult); + } + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseVectorMatch.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseVectorMatch.java new file mode 100755 index 0000000000..5df30dce9d --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/ResponseVectorMatch.java @@ -0,0 +1,58 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.DescriptionManager; +import ghidra.xml.XmlPullParser; + +/** + * Response to a request for functions with specific vector ids + * Full ExecutableRecords and FunctionDescriptions are instantiated in this object's DescriptionManager + */ +public class ResponseVectorMatch extends QueryResponseRecord { + + public DescriptionManager manage; // Set of functions (and executables) matching vector id request + + public ResponseVectorMatch() { + super("responsevectormatch"); + manage = new DescriptionManager(); + } + + @Override + public DescriptionManager getDescriptionManager() { + return manage; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + fwrite.append('<').append(name).append(">\n"); + manage.saveXml(fwrite); + fwrite.append("\n"); + } + + @Override + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory) + throws LSHException { + parser.start(name); + manage.restoreXml(parser, vectorFactory); + parser.end(); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityNote.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityNote.java new file mode 100755 index 0000000000..4183ce5cab --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityNote.java @@ -0,0 +1,91 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.Map; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * A description of a single function match + * + */ +public class SimilarityNote implements Comparable { + private FunctionDescription func; // Description of function that was matched + private double sim; // Similarity of the match + private double signif; // Significance of the match + + public SimilarityNote() {} // For use with restoreXml + + public SimilarityNote(FunctionDescription f,double sm,double sf) { + func = f; + sim = sm; + signif = sf; + } + + public FunctionDescription getFunctionDescription() { return func; } + + public double getSimilarity() { return sim; } + + public double getSignificance() { return signif; } + + public void transfer(DescriptionManager manage,boolean transsig) throws LSHException { + func = manage.transferFunction(func,transsig); + } + + public void setTransfer(SimilarityNote op2,DescriptionManager manage,boolean transsig) throws LSHException { + func = manage.transferFunction(op2.func, transsig); + sim = op2.sim; + signif = op2.signif; + } + + public void saveXml(Writer write) throws IOException { + StringBuilder buf = new StringBuilder(); + buf.append("\n"); + buf.append(" ").append(Double.toString(sim)).append("\n"); + buf.append(" ").append(Double.toString(signif)).append("\n"); + buf.append("\n"); + write.append(buf.toString()); + } + + public void restoreXml(XmlPullParser parser,DescriptionManager manage, Map exeMap) throws LSHException { + XmlElement el = parser.start("note"); + int id = SpecXmlUtils.decodeInt(el.getAttribute("id")); + ExecutableRecord exe = exeMap.get(id); + long address = SpecXmlUtils.decodeLong(el.getAttribute("addr")); + func = manage.findFunction(el.getAttribute("name"), address, exe); + parser.start("sim"); + sim = Double.parseDouble(parser.end().getText()); + parser.start("sig"); + signif = Double.parseDouble(parser.end().getText()); + parser.end(); + } + + @Override + public int compareTo(SimilarityNote o) { + return func.compareTo(o.func); + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityResult.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityResult.java new file mode 100755 index 0000000000..9e630a9a58 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityResult.java @@ -0,0 +1,135 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * A collection of matches to an (originally) queried function + * + */ +public class SimilarityResult implements Iterable { + private FunctionDescription basefunc; // The original function that was queried + private List notes; // Functions to which base is similar + private int totalcount; // Total number of functions in database meeting similarity and significance + + public SimilarityResult() { + } // For use with restoreXml + + public SimilarityResult(FunctionDescription f) { + basefunc = f; + notes = new ArrayList(); + totalcount = 0; + } + + public void addNote(FunctionDescription f, double similarity, double significance) { + notes.add(new SimilarityNote(f, similarity, significance)); + } + + public FunctionDescription getBase() { + return basefunc; + } + + public int size() { + return notes.size(); + } + + public void setTotalCount(int count) { + totalcount = count; + } + + public int getTotalCount() { + return totalcount; + } + + @Override + public Iterator iterator() { + return notes.iterator(); + } + + public void transfer(DescriptionManager manage, boolean transsig) throws LSHException { + for (SimilarityNote note : notes) { + note.transfer(manage, transsig); + } + } + + public void setTransfer(SimilarityResult op2, DescriptionManager qmanage, + DescriptionManager rmanage, boolean transsig) throws LSHException { + ExecutableRecord erec = qmanage.findExecutable(op2.basefunc.getExecutableRecord().getMd5()); + basefunc = + qmanage.findFunction(op2.basefunc.getFunctionName(), op2.basefunc.getAddress(), erec); + totalcount = op2.totalcount; + notes = new ArrayList(); + for (SimilarityNote item : op2.notes) { + SimilarityNote newitem = new SimilarityNote(); + newitem.setTransfer(item, rmanage, transsig); + notes.add(newitem); + } + } + + public void saveXml(Writer write) throws IOException { + StringBuilder buf = new StringBuilder(); + buf.append("\n"); + write.append(buf.toString()); + Iterator iter = notes.iterator(); + while (iter.hasNext()) { + iter.next().saveXml(write); + } + write.append("\n"); + } + + public void restoreXml(XmlPullParser parser, DescriptionManager qmanage, + DescriptionManager rmanage, + Map qMap, Map rMap) + throws LSHException { + notes = new ArrayList(); + XmlElement el = parser.start("simres"); + int id = SpecXmlUtils.decodeInt(el.getAttribute("id")); + ExecutableRecord exe = qMap.get(id); + long address = SpecXmlUtils.decodeLong(el.getAttribute("addr")); + basefunc = qmanage.findFunction(el.getAttribute("name"), address, exe); + totalcount = SpecXmlUtils.decodeInt(el.getAttribute("total")); + while (parser.peek().isStart()) { + SimilarityNote newnote = new SimilarityNote(); + newnote.restoreXml(parser, rmanage, rMap); + notes.add(newnote); + } + parser.end(); + } + + public void sortNotes() { + Collections.sort(notes); + } + + @Override + public String toString() { + return getClass().getSimpleName() + " - base function: " + basefunc; + } +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityVectorResult.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityVectorResult.java new file mode 100755 index 0000000000..126fd825b2 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/SimilarityVectorResult.java @@ -0,0 +1,112 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.LSHException; +import ghidra.features.bsim.query.description.*; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.XmlElement; +import ghidra.xml.XmlPullParser; + +/** + * A collection of vector matches to an (originally) queried function + * + */ +public class SimilarityVectorResult { + private FunctionDescription basefunc; // The original function that was queried + private List notes; // Vectors to which base is similar + private int totalcount; // Total number of functions in database matching one of these vectors + + public SimilarityVectorResult() { + // For use with restoreXml + } + + public SimilarityVectorResult(FunctionDescription f) { + basefunc = f; + notes = new ArrayList(); + totalcount = 0; + } + + public void addNotes(List newnotes) { + for (VectorResult note : newnotes) { + totalcount += note.hitcount; + notes.add(note); + } + } + + public Iterator iterator() { + return notes.iterator(); + } + + public FunctionDescription getBase() { + return basefunc; + } + + public int getTotalCount() { + return totalcount; + } + + public void sortNotes() { + Collections.sort(notes, new Comparator() { + + @Override + public int compare(VectorResult o1, VectorResult o2) { + return Long.compare(o1.vectorid, o2.vectorid); + } + + }); + } + + public void saveXml(Writer write) throws IOException { + StringBuilder buf = new StringBuilder(); + buf.append("\n"); + write.append(buf.toString()); + Iterator iter = notes.iterator(); + while (iter.hasNext()) { + iter.next().saveXml(write); + } + write.append("\n"); + } + + public void restoreXml(XmlPullParser parser, LSHVectorFactory vectorFactory, + DescriptionManager qmanage, Map exeMap) throws LSHException { + notes = new ArrayList(); + XmlElement el = parser.start("simvecres"); + int id = SpecXmlUtils.decodeInt(el.getAttribute("id")); + ExecutableRecord exe = exeMap.get(id); + long address = SpecXmlUtils.decodeLong(el.getAttribute("addr")); + basefunc = qmanage.findFunction(el.getAttribute("name"), address, exe); + totalcount = 0; + while (parser.peek().isStart()) { + VectorResult newnote = new VectorResult(); + newnote.restoreXml(parser, vectorFactory); + notes.add(newnote); + totalcount += newnote.hitcount; + } + parser.end(); + } + +} diff --git a/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/StagingManager.java b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/StagingManager.java new file mode 100755 index 0000000000..bd84d1012b --- /dev/null +++ b/Ghidra/Features/BSim/src/main/java/ghidra/features/bsim/query/protocol/StagingManager.java @@ -0,0 +1,63 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.protocol; + +import ghidra.features.bsim.query.LSHException; + +/** + * Abstract class for splitting up a (presumably large) query into smaller pieces + * The object must be configured by a call to setQuery with details of the staged query + * and then typically a call to setGlobalManager which specifies the data for the whole query + * + * Placing the actual staged queries is accomplished by first calling initialize, + * which establishes the first stage query, obtainable via the getQuery method. + * Successive stage queries are built by calling nextStage repeatedly until it returns false. + * + */ +public abstract class StagingManager { + protected BSimQuery globalQuery; // The global query + protected int totalsize; // Total number of separate queries being staged + protected int queriesmade; // Number of queries currently sent + + public int getTotalSize() { + return totalsize; + } + + public int getQueriesMade() { + return queriesmade; + } + + /** + * Get the current staged query + * @return the QueryResponseRecord object + */ + public abstract BSimQuery getQuery(); + + /** + * Establish the first query stage + * @param q the query + * @return true if the initial query is constructed + * @throws LSHException if the initialization fails + */ + public abstract boolean initialize(BSimQuery q) throws LSHException; + + /** + * Establish the next query stage + * @return true if a next query is constructed + * @throws LSHException if creating the new query fails + */ + public abstract boolean nextStage() throws LSHException; +} diff --git a/Ghidra/Features/BSim/src/main/resources/bsim.log4j.xml b/Ghidra/Features/BSim/src/main/resources/bsim.log4j.xml new file mode 100644 index 0000000000..88d16dfbbd --- /dev/null +++ b/Ghidra/Features/BSim/src/main/resources/bsim.log4j.xml @@ -0,0 +1,33 @@ + + + + + + + + + + + + + + + + + + + + + + + + + + + diff --git a/Ghidra/Features/BSim/src/main/resources/images/checkmark_yellow.gif b/Ghidra/Features/BSim/src/main/resources/images/checkmark_yellow.gif new file mode 100644 index 0000000000..91be11ad28 Binary files /dev/null and b/Ghidra/Features/BSim/src/main/resources/images/checkmark_yellow.gif differ diff --git a/Ghidra/Features/BSim/src/main/resources/images/flag_green.png b/Ghidra/Features/BSim/src/main/resources/images/flag_green.png new file mode 100644 index 0000000000..e4bc611f87 Binary files /dev/null and b/Ghidra/Features/BSim/src/main/resources/images/flag_green.png differ diff --git a/Ghidra/Features/BSim/src/main/resources/images/preferences-desktop-user-password.png b/Ghidra/Features/BSim/src/main/resources/images/preferences-desktop-user-password.png new file mode 100644 index 0000000000..6114a19ecd Binary files /dev/null and b/Ghidra/Features/BSim/src/main/resources/images/preferences-desktop-user-password.png differ diff --git a/Ghidra/Features/BSim/src/main/resources/images/preferences-web-browser-shortcuts-32.png b/Ghidra/Features/BSim/src/main/resources/images/preferences-web-browser-shortcuts-32.png new file mode 100644 index 0000000000..9798a9ad3d Binary files /dev/null and b/Ghidra/Features/BSim/src/main/resources/images/preferences-web-browser-shortcuts-32.png differ diff --git a/Ghidra/Features/BSim/src/main/resources/images/preferences-web-browser-shortcuts.png b/Ghidra/Features/BSim/src/main/resources/images/preferences-web-browser-shortcuts.png new file mode 100755 index 0000000000..c694395dad Binary files /dev/null and b/Ghidra/Features/BSim/src/main/resources/images/preferences-web-browser-shortcuts.png differ diff --git a/Ghidra/Features/BSim/src/main/resources/images/view_top_bottom.png b/Ghidra/Features/BSim/src/main/resources/images/view_top_bottom.png new file mode 100644 index 0000000000..2f1f06aecb Binary files /dev/null and b/Ghidra/Features/BSim/src/main/resources/images/view_top_bottom.png differ diff --git a/Ghidra/Features/BSim/src/main/resources/log4j-appender-console.xml b/Ghidra/Features/BSim/src/main/resources/log4j-appender-console.xml new file mode 100644 index 0000000000..f02f948217 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/resources/log4j-appender-console.xml @@ -0,0 +1,14 @@ + + + + + + + + + + + + diff --git a/Ghidra/Features/BSim/src/main/resources/log4j-appender-rolling-file.xml b/Ghidra/Features/BSim/src/main/resources/log4j-appender-rolling-file.xml new file mode 100644 index 0000000000..f9cf8a6284 --- /dev/null +++ b/Ghidra/Features/BSim/src/main/resources/log4j-appender-rolling-file.xml @@ -0,0 +1,13 @@ + + + + + + + + + + diff --git a/Ghidra/Features/BSim/src/screen/java/help/screenshot/BSimSearchPluginScreenShots.java b/Ghidra/Features/BSim/src/screen/java/help/screenshot/BSimSearchPluginScreenShots.java new file mode 100755 index 0000000000..98244f090c --- /dev/null +++ b/Ghidra/Features/BSim/src/screen/java/help/screenshot/BSimSearchPluginScreenShots.java @@ -0,0 +1,313 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package help.screenshot; + +import java.util.*; + +import org.junit.Before; +import org.junit.Test; + +import docking.DockingWindowManager; +import docking.action.DockingActionIf; +import ghidra.app.services.ProgramManager; +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.BSimSearchPluginTestHelper; +import ghidra.features.bsim.gui.overview.BSimOverviewProvider; +import ghidra.features.bsim.gui.overview.BSimOverviewTestHelper; +import ghidra.features.bsim.gui.search.dialog.*; +import ghidra.features.bsim.gui.search.results.*; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.facade.*; +import ghidra.features.bsim.query.protocol.ResponseNearest; +import ghidra.features.bsim.query.protocol.ResponseNearestVector; +import ghidra.program.database.ProgramDB; +import ghidra.program.model.address.Address; +import ghidra.program.model.address.AddressFormatException; +import ghidra.test.ToyProgramBuilder; +import ghidra.util.Msg; + +/** + * Captures screenshots for the BSim help pages. Note that this class does not + * generate ALL images used in the BSim help. The following are generated + * by hand, but should be addressed to be created here at some point: + * + * functionmatch.png + * toolbar.png + * client.png + * + */ +public class BSimSearchPluginScreenShots extends GhidraScreenShotGenerator { + protected static final String FUN1_ADDR = "0x01001100"; + protected static final String FUN2_ADDR = "0x01001200"; + protected static final String FUN3_ADDR = "0x01001300"; + protected static final String FUN4_ADDR = "0x01001400"; + + private BSimSearchPlugin plugin; + + public BSimSearchPluginScreenShots() { + super(); + } + + @Override + @Before + public void setUp() throws Exception { + super.setUp(); + tool.addPlugin(BSimSearchPlugin.class.getName()); + + plugin = getPlugin(tool, BSimSearchPlugin.class); + goTo(tool, program, FUN1_ADDR); + removeAllServers(); + } + + @Override + public void loadProgram() throws Exception { + program = buildProgram(); + runSwing(() -> { + ProgramManager pm = tool.getService(ProgramManager.class); + pm.openProgram(program.getDomainFile()); + }); + } + + @Override + public void dockingSetUp() { + // We get an error dialog about default tools, since this test is not in the + // Integration Test project. Disable the error dialogs. + setErrorGUIEnabled(false); + } + + @Test + public void testManageServersDialog() { + addTestServer(new BSimServerInfo(DBType.postgres, "100.50.123.5", 123, "testDB")); + addTestServer(new BSimServerInfo(DBType.postgres, "100.50.123.5", 134, "anotherDB")); + addTestServer(new BSimServerInfo(DBType.file, "100.50.123.5", 134, "/bsim/database1")); + + DockingActionIf action = getAction(plugin, "Manage BSim Servers"); + performAction(action, false); + BSimServerDialog dialog = waitForDialogComponent(BSimServerDialog.class); + + captureDialog(dialog); + dialog.close(); + } + + @Test + public void testAddServerDialog() { + CreateBsimServerInfoDialog dialog = new CreateBsimServerInfoDialog(); + runSwingLater(() -> DockingWindowManager.showDialog(dialog)); + waitForSwing(); + captureDialog(dialog); + dialog.close(); + } + + @Test + public void testBSimOverviewDialog() { + + DockingActionIf action = getAction(plugin, "BSim Overview"); + performAction(action, false); + + BSimOverviewDialog dialog = waitForDialogComponent(BSimOverviewDialog.class); + + captureDialog(dialog); + dialog.close(); + } + + @Test + public void testBSimOverviewResults() { + DockingActionIf action = getAction(plugin, "BSim Overview"); + performAction(action, false); + + BSimOverviewDialog dialog = waitForDialogComponent(BSimOverviewDialog.class); + + FunctionDatabaseTestDouble database = createTestDoubleWithOverviewResults(); + BSimOverviewTestHelper.setBSimOVerviewTestServer(plugin, dialog, database); + + pressButtonByText(dialog, "Overview"); + waitForComponentProvider(BSimOverviewProvider.class); + captureIsolatedProvider(BSimOverviewProvider.class, 600, 300); + } + + @Test + public void testBSimSearchDialog() { + + goTo(tool, program, FUN1_ADDR); + + DockingActionIf action = getAction(plugin, "BSim Search Functions"); + performAction(action, false); + + BSimSearchDialog dialog = waitForDialogComponent(BSimSearchDialog.class); + +// urlEntryField = (JTextField) getInstanceField("bsimFilterPanel", bsimSearchDialog); +// urlEntryField.setText(DB_URL); + +// BSimFilterPanel filterPanel = getInstanceField("bsimFilterPanel", bsimSearchDialog); +// +// filterPanel = (DatabaseFilterPanel) getInstanceField("filterPanel", bsimSearchDialog); +// filterPanel.setSpecificFilter("archequals", "x86:LE:32:default"); + + captureDialog(dialog); + dialog.close(); + } + + @Test + public void testBSimResultsProvider() { + goTo(tool, program, FUN1_ADDR); + + DockingActionIf action = getAction(plugin, "BSim Search Functions"); + performAction(action, false); + + BSimSearchDialog dialog = waitForDialogComponent(BSimSearchDialog.class); + + FunctionDatabaseTestDouble database = createTestFunctionDatabaseTestDouble(); + BSimSearchDialogTestHelper.setBSimSearchTestServer(plugin, dialog, database); + + pressButtonByText(dialog, "Search"); + + waitForComponentProvider(BSimSearchResultsProvider.class); + captureIsolatedProvider(BSimSearchResultsProvider.class, 800, 500); + + } + + /* + * Showing the results pane using a normal workflow is a bit tricky, and ultimately not + * worth the effort. To generate this panel we can just fake out the data and show + * the window directly. + */ + @Test + public void testApplyResultsPanel() throws Exception { + + List results = new ArrayList<>(); + BSimApplyResult r1 = new BSimApplyResult("fun_0001", "foo1", BSimResultStatus.ERROR, + addr("01001100"), "ERROR: Attempting to apply multiple names to the same function."); + BSimApplyResult r2 = new BSimApplyResult("fun_0001", "foo2", BSimResultStatus.ERROR, + addr("0x01001100"), "ERROR: Attempting to apply multiple names to the same function."); + BSimApplyResult r3 = new BSimApplyResult("set_string", "_set_string", + BSimResultStatus.NAME_APPLIED, addr("0x01001100"), ""); + BSimApplyResult r4 = new BSimApplyResult("add_code", "addcode2", + BSimResultStatus.SIGNATURE_APPLIED, addr("0x01001100"), ""); + BSimApplyResult r5 = new BSimApplyResult("__libc_csu_fini", "__libc_csu_fini", + BSimResultStatus.IGNORED, addr("0x01001100"), "INFO: No change. Names are the same."); + results.add(r1); + results.add(r2); + results.add(r3); + results.add(r4); + results.add(r5); + + BSimApplyResultsDisplayDialog resultsPanel = + runSwing(() -> new BSimApplyResultsDisplayDialog(plugin.getTool(), results, program)); + + resultsPanel.setPreferredSize(900, 300); + + tool.showDialog(resultsPanel); + waitForDialogComponent(BSimApplyResultsDisplayDialog.class); + captureDialog(resultsPanel); + } + +//================================================================================================== +// Private Methods +//================================================================================================== + + private Address addr(String address) { + try { + return program.getAddressFactory().getDefaultAddressSpace().getAddress(address); + } + catch (AddressFormatException e) { + Msg.error(this, "Error converting " + address + " to an Address", e); + } + + return null; + } + + protected void initializeTool() throws Exception { + + plugin = env.getPlugin(BSimSearchPlugin.class); + + program = buildProgram(); + ProgramManager pm = tool.getService(ProgramManager.class); + pm.openProgram(program.getDomainFile()); + + showTool(tool); + + } + + private ProgramDB buildProgram() throws Exception { + ToyProgramBuilder builder = new ToyProgramBuilder("Sample Program", true); + + builder.createMemory(".text", "0x1001000", 0x10000); + builder.createReturnInstruction(FUN1_ADDR); + builder.createEmptyFunction(null, FUN1_ADDR, 10, null); + builder.createReturnInstruction(FUN2_ADDR); + builder.createEmptyFunction(null, FUN2_ADDR, 10, null); + builder.createReturnInstruction(FUN3_ADDR); + builder.createEmptyFunction(null, FUN3_ADDR, 10, null); + builder.createReturnInstruction(FUN4_ADDR); + builder.createEmptyFunction(null, FUN4_ADDR, 10, null); + return builder.getProgram(); + } + + private FunctionDatabaseTestDouble createTestFunctionDatabaseTestDouble() { + FunctionDatabaseTestDouble database = new FunctionDatabaseTestDouble(); + + // create some canned data + ResponseNearest response = new ResponseNearest(null); + response.result.add(new TestSimilarityResult("queryFunction", "exec1", "matchFunction1", + 0x01001100, 0.9d, 15.0d)); + response.result.add(new TestSimilarityResult("queryFunction", "exec2", "matchFunction2", + 0x01001100, 0.9d, 15.0d)); + response.result.add(new TestSimilarityResult("queryFunction", "exec1", "matchFunction3", + 0x01001100, 0.9d, 15.0d)); + response.result.add(new TestSimilarityResult("queryFunction", "exec1", "matchFunction4", + 0x01001100, 0.9d, 15.0d)); + + database.setQueryResponse(response); // set a valid response to be returned on query + database.setCanInitialize(true); // initialize may be called--this is OK + + return database; + } + + private FunctionDatabaseTestDouble createTestDoubleWithOverviewResults() { + FunctionDatabaseTestDouble database = new FunctionDatabaseTestDouble(); + // create some canned data + ResponseNearestVector response = new ResponseNearestVector(null); + response.result.add(new TestNearestVectorResult("function1", "exec1", 12, 0.9d)); + response.result.add(new TestNearestVectorResult("function1", "exec2", 8, .9d)); + response.result.add(new TestNearestVectorResult("function2", "exec3", 32, .9d)); + response.result.add(new TestNearestVectorResult("function3", "exec4", 4, 9d)); + + database.setQueryResponse(response); // set a valid response to be returned on query + database.setCanInitialize(true); // initialize may be called--this is OK + + return database; + } + + private void addTestServer(BSimServerInfo serverInfo) { + runSwing(() -> { + BSimServerManager serverManager = BSimSearchPluginTestHelper.getServerManager(plugin); + serverManager.addServer(serverInfo); + }); + + } + + private void removeAllServers() { + runSwing(() -> { + BSimServerManager serverManager = BSimSearchPluginTestHelper.getServerManager(plugin); + Set serverInfos = serverManager.getServerInfos(); + for (BSimServerInfo info : serverInfos) { + serverManager.removeServer(info, true); + } + }); + + } +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/AbstractBSimPluginTest.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/AbstractBSimPluginTest.java new file mode 100644 index 0000000000..5322621000 --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/AbstractBSimPluginTest.java @@ -0,0 +1,157 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui; + +import org.junit.After; +import org.junit.Before; + +import docking.DockingErrorDisplay; +import docking.action.DockingActionIf; +import ghidra.app.events.ProgramLocationPluginEvent; +import ghidra.app.plugin.core.codebrowser.CodeBrowserPlugin; +import ghidra.app.services.ProgramManager; +import ghidra.features.bsim.gui.overview.*; +import ghidra.features.bsim.gui.search.dialog.BSimOverviewDialog; +import ghidra.features.bsim.gui.search.dialog.BSimSearchDialog; +import ghidra.features.bsim.gui.search.results.*; +import ghidra.framework.plugintool.PluginTool; +import ghidra.program.database.ProgramDB; +import ghidra.program.model.address.Address; +import ghidra.program.model.address.AddressFactory; +import ghidra.program.util.ProgramLocation; +import ghidra.test.*; +import ghidra.util.Msg; + +public class AbstractBSimPluginTest extends AbstractGhidraHeadedIntegrationTest { + + protected static final String FUN1_ADDR = "0x01001100"; + protected static final String FUN2_ADDR = "0x01001200"; + protected static final String FUN3_ADDR = "0x01001300"; + protected TestEnv env; + protected PluginTool tool; + protected ProgramDB program; + + protected BSimSearchPlugin searchPlugin; + protected BSimSearchDialog searchDialog; + protected DockingActionIf searchAction; + protected DockingActionIf overviewAction; + protected BSimSearchResultsProvider resultsProvider; + protected BSimOverviewDialog overviewDialog; + private BSimOverviewProvider overviewProvider; + + @Before + public void setUp() throws Exception { + + setErrorGUIEnabled(false); + + env = new TestEnv(); + tool = env.getTool(); + + initializeTool(); + + // this will allow our expected warning dialogs to display + Msg.setErrorDisplay(new DockingErrorDisplay()); + } + + @After + public void tearDown() throws Exception { + env.release(program); + env.dispose(); + } + + protected void initializeTool() throws Exception { + tool.addPlugin(CodeBrowserPlugin.class.getName()); + tool.addPlugin(BSimSearchPlugin.class.getName()); + + searchPlugin = env.getPlugin(BSimSearchPlugin.class); + + program = buildProgram(); + ProgramManager pm = tool.getService(ProgramManager.class); + pm.openProgram(program.getDomainFile()); + + showTool(tool); + + searchAction = getAction(searchPlugin, "BSim Search Functions"); + overviewAction = getAction(searchPlugin, "BSim Overview"); + } + + private ProgramDB buildProgram() throws Exception { + ToyProgramBuilder builder = new ToyProgramBuilder("notepad", true); + + builder.createMemory(".text", "0x1001000", 0x10000); + builder.createReturnInstruction(FUN1_ADDR); + builder.createEmptyFunction(null, FUN1_ADDR, 10, null); + builder.createReturnInstruction(FUN2_ADDR); + builder.createEmptyFunction(null, FUN2_ADDR, 10, null); + builder.createReturnInstruction(FUN3_ADDR); + builder.createEmptyFunction(null, FUN3_ADDR, 10, null); + return builder.getProgram(); + } + + protected void doSearch() { + pressButtonByText(searchDialog, "Search"); + resultsProvider = waitForComponentProvider(BSimSearchResultsProvider.class); + BSimMatchResultsModel matchesModel = getMatchesModel(); + waitForTableModel(matchesModel); + } + + protected void doOverview() { + pressButtonByText(overviewDialog, "Overview"); + overviewProvider = waitForComponentProvider(BSimOverviewProvider.class); + BSimOverviewModel overviewModel = getOverviewModel(); + waitForTableModel(overviewModel); + } + + protected void invokeBSimSearchAction() { + performAction(searchAction, false); + + searchDialog = waitForDialogComponent(BSimSearchDialog.class); + + } + + protected void invokeBSimOverviewAction() { + performAction(overviewAction, false); + + overviewDialog = waitForDialogComponent(BSimOverviewDialog.class); + + } + + protected Address addr(String addressString) { + AddressFactory factory = program.getAddressFactory(); + return factory.getAddress(addressString); + } + + protected void goTo(String addr) { + ProgramLocation location = new ProgramLocation(program, addr(addr)); + tool.firePluginEvent(new ProgramLocationPluginEvent("test", location, program)); + waitForSwing(); + program.flushEvents(); + waitForSwing(); + } + + protected BSimMatchResultsModel getMatchesModel() { + return BSimSearchResultsTestHelper.getSearchResultsModel(resultsProvider); + } + + protected BSimOverviewModel getOverviewModel() { + return BSimOverviewTestHelper.getOverviewModel(overviewProvider); + } + + protected BSimExecutablesSummaryModel getExecutablesModel() { + return BSimSearchResultsTestHelper.getExecutablesModel(resultsProvider); + } + +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/BSimSearchPluginTest.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/BSimSearchPluginTest.java new file mode 100755 index 0000000000..4523a79ff1 --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/BSimSearchPluginTest.java @@ -0,0 +1,191 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui; + +import static org.junit.Assert.*; + +import java.util.Set; + +import org.junit.Test; + +import docking.widgets.OkDialog; +import ghidra.app.events.ProgramSelectionPluginEvent; +import ghidra.features.bsim.gui.overview.BSimOverviewTestHelper; +import ghidra.features.bsim.gui.search.dialog.BSimSearchDialogTestHelper; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.facade.*; +import ghidra.features.bsim.query.protocol.ResponseNearest; +import ghidra.features.bsim.query.protocol.ResponseNearestVector; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.util.ProgramSelection; + +public class BSimSearchPluginTest extends AbstractBSimPluginTest { + + @Test + public void testNoFunctionSelected() { + performAction(searchAction, false); + OkDialog dialog = waitForDialogComponent(OkDialog.class); + assertNotNull(dialog); + } + + @Test + public void testSingleFunctionQuery() { + goTo(FUN1_ADDR); + invokeBSimSearchAction(); + Set functions = + BSimSearchDialogTestHelper.getSelectedFunctions(searchDialog); + assertEquals(1, functions.size()); + assertEquals("FUN_01001100", functions.iterator().next().getName()); + + assertNull(getServer()); + FunctionDatabaseTestDouble database = createTestDoubleWithDataForSingleFunction(); + BSimSearchDialogTestHelper.setBSimSearchTestServer(searchPlugin, searchDialog, database); + + doSearch(); + + assertEquals(4, getMatchesModel().getRowCount()); + assertEquals(2, getExecutablesModel().getRowCount()); + } + + @Test + public void testRepeatedQueryRemembersSettings() { + goTo(FUN1_ADDR); + invokeBSimSearchAction(); + assertNull(getServer()); + FunctionDatabaseTestDouble database = createTestDoubleWithDataForSingleFunction(); + BSimSearchDialogTestHelper.setBSimSearchTestServer(searchPlugin, searchDialog, database); + + doSearch(); + + invokeBSimSearchAction(); + assertNotNull(getServer()); + + } + + @Test + public void testQueryWithSelection() { + createMultiFunctionSelection(); + invokeBSimSearchAction(); + + Set functions = + BSimSearchDialogTestHelper.getSelectedFunctions(searchDialog); + assertEquals(3, functions.size()); + + FunctionDatabaseTestDouble database = createTestDoubleWithDataForSingleFunction(); + BSimSearchDialogTestHelper.setBSimSearchTestServer(searchPlugin, searchDialog, database); + + doSearch(); + assertEquals(4, getMatchesModel().getRowCount()); + assertEquals(2, getExecutablesModel().getRowCount()); + } + + @Test + public void testOverviewQuery() { + invokeBSimOverviewAction(); + FunctionDatabaseTestDouble database = createTestDoubleWithOverviewResults(); + BSimOverviewTestHelper.setBSimOVerviewTestServer(searchPlugin, overviewDialog, database); + + doOverview(); + assertEquals(4, getOverviewModel().getRowCount()); + } + +// private void setBSimTestServer() { +// runSwing(() -> { +// FunctionDatabaseTestDouble database = +// createTestDoubleWithDataForSingleFunction(TEST_URL); +// searchPlugin.setQueryServiceFactory(new TestSFQueryServiceFactory(database)); +// BSimServerInfo info = new TestBSimServerInfo(database); +// BSimServerManager serverManager = searchPlugin.getServerManager(); +// serverManager.addServer(info); +// BSimSearchDialogTestHelper.setSelectedServer(dialog, server); +// }); +// } + +// private void setBSimOVerviewTestServer() { +// runSwing(() -> { +// FunctionDatabaseTestDouble database = createTestDoubleWithOverviewResults(TEST_URL); +// searchPlugin.setQueryServiceFactory(new TestSFQueryServiceFactory(database)); +// BSimServerInfo info = new TestBSimServerInfo(database); +// BSimServerManager serverManager = searchPlugin.getServerManager(); +// serverManager.addServer(info); +// BSimSearchDialogTestHelper.setSelectedServer(dialog, server); +// }); +// } + + private FunctionDatabaseTestDouble createTestDoubleWithDataForSingleFunction() { + FunctionDatabaseTestDouble database = new FunctionDatabaseTestDouble(); + + // create some canned data + ResponseNearest response = new ResponseNearest(null); + response.result.add(new TestSimilarityResult("queryFunction", "exec1", "matchFunction1", + 01001100, 0.9d, 15.0d)); + response.result.add(new TestSimilarityResult("queryFunction", "exec2", "matchFunction2", + 01001100, 0.9d, 15.0d)); + response.result.add(new TestSimilarityResult("queryFunction", "exec1", "matchFunction3", + 01001100, 0.9d, 15.0d)); + response.result.add(new TestSimilarityResult("queryFunction", "exec1", "matchFunction4", + 01001100, 0.9d, 15.0d)); + + database.setQueryResponse(response); // set a valid response to be returned on query + database.setCanInitialize(true); // initialize may be called--this is OK + + return database; + } + + private FunctionDatabaseTestDouble createTestDoubleWithOverviewResults() { + FunctionDatabaseTestDouble database = new FunctionDatabaseTestDouble(); + // create some canned data + ResponseNearestVector response = new ResponseNearestVector(null); + response.result.add(new TestNearestVectorResult("function1", "exec1", 12, 0.9d)); + response.result.add(new TestNearestVectorResult("function1", "exec2", 8, .9d)); + response.result.add(new TestNearestVectorResult("function2", "exec3", 32, .9d)); + response.result.add(new TestNearestVectorResult("function3", "exec4", 4, 9d)); + + database.setQueryResponse(response); // set a valid response to be returned on query + database.setCanInitialize(true); // initialize may be called--this is OK + + return database; + } + + protected void createMultiFunctionSelection() { + ProgramSelection selection = new ProgramSelection(addr(FUN1_ADDR), addr(FUN3_ADDR)); + tool.firePluginEvent(new ProgramSelectionPluginEvent("test", selection, program)); + waitForSwing(); + program.flushEvents(); + waitForSwing(); + } + + private BSimServerInfo getServer() { + return BSimSearchDialogTestHelper.getSelectedServer(searchDialog); + } + + class TestBSimServerInfo extends BSimServerInfo { + + private FunctionDatabase database; + + public TestBSimServerInfo(FunctionDatabase database) { + super(DBType.postgres, "0.0.0.0", 123, "testDB"); + this.database = database; + } + + @Override + public FunctionDatabase getFunctionDatabase(boolean async) { + return database; + } + } + +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/BSimSearchPluginTestHelper.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/BSimSearchPluginTestHelper.java new file mode 100644 index 0000000000..c80999009e --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/BSimSearchPluginTestHelper.java @@ -0,0 +1,30 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui; + +import ghidra.features.bsim.gui.search.dialog.BSimServerManager; +import ghidra.features.bsim.query.facade.SFQueryServiceFactory; + +public class BSimSearchPluginTestHelper { + public static BSimServerManager getServerManager(BSimSearchPlugin plugin) { + return plugin.getServerManager(); + } + + public static void setQueryServiceFactory(BSimSearchPlugin plugin, + SFQueryServiceFactory factory) { + plugin.setQueryServiceFactory(factory); + } +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/QueryFilterTest.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/QueryFilterTest.java new file mode 100755 index 0000000000..72ffd6a255 --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/QueryFilterTest.java @@ -0,0 +1,95 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui; + +import static org.junit.Assert.*; + +import java.sql.SQLException; +import java.util.List; + +import org.junit.Test; + +import ghidra.features.bsim.gui.filters.*; +import ghidra.features.bsim.gui.search.dialog.*; +import ghidra.features.bsim.query.client.*; +import ghidra.features.bsim.query.protocol.BSimFilter; +import ghidra.util.exception.AssertException; + +/** + * Tests the ability of the filter validators to correctly parse inputs. + * + */ +public class QueryFilterTest extends AbstractBSimPluginTest { + + private BSimFilterPanel filterPanel; + + /** + * Tests that we can construct a proper SQL statement using the filters on the BSIM query panel. + * + * To do this we open the query dialog, add a few filters, populate them with several filters, then + * call the appropriate functions in the ExecutableFilter class that generate the SLQ string, and + * compare it to what we expect. + */ + @Test + public void sqlConstructionTest() { + + // This is the SQL statement we want to produce. + final String SQL_TRUTH = + "AND (desctable.id_exe = exetable.id) AND (exetable.name_exec != 'bad exec name' AND exetable.name_exec != 'bad exec name 2') AND (exetable.name_exec = 'exec name 1' OR exetable.name_exec = 'exec name 3')"; + + BSimFilterSet filterSet = new BSimFilterSet(); + filterSet.addEntry(new ExecutableNameBSimFilterType(), List.of("exec name 1")); + filterSet.addEntry(new Md5BSimFilterType(), List.of("0x0123456789")); + filterSet.addEntry(new NotExecutableNameBSimFilterType(), + List.of("bad exec name", "bad exec name 2")); + filterSet.addEntry(new DateEarlierBSimFilterType(""), List.of("Jan 01, 2000")); + filterSet.addEntry(new ExecutableNameBSimFilterType(), List.of("exec name 3")); + + runSwing(() -> filterPanel.setFilterSet(filterSet)); + // Now go ahead and generate the SQL and verify it's correct. + IDSQLResolution[] ids = new IDSQLResolution[] { null, null, null, null, null }; + String sql = runSwing(() -> generateSQL(ids)); + assertEquals(SQL_TRUTH, sql); + } + + protected void initializeTool() throws Exception { + super.initializeTool(); + goTo(FUN1_ADDR); + performAction(searchAction, false); + + searchDialog = waitForDialogComponent(BSimSearchDialog.class); + filterPanel = BSimSearchDialogTestHelper.getFilterPanel(searchDialog); + } + + /** + * Generates the SQL that will be used in the query, based on the current + * filter settings. + * + * @param ids resolution IDs + * @return the query string + * @throws SQLException if there is a problem creating the filter + */ + private String generateSQL(IDSQLResolution[] ids) { + try { + BSimFilterSet filterSet = filterPanel.getFilterSet(); + BSimFilter filter = filterSet.getBSimFilter(); + BSimSqlClause sql = SQLEffects.createFilter(filter, ids, null); + return sql.whereClause().trim(); + } catch (SQLException e) { + throw new AssertException(e); + } + } +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/overview/BSimOverviewTestHelper.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/overview/BSimOverviewTestHelper.java new file mode 100644 index 0000000000..7be8bb04ff --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/overview/BSimOverviewTestHelper.java @@ -0,0 +1,46 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.overview; + +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.BSimSearchPluginTestHelper; +import ghidra.features.bsim.gui.search.dialog.*; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.facade.TestBSimServerInfo; +import ghidra.features.bsim.query.facade.TestSFQueryServiceFactory; +import ghidra.util.Swing; + +public class BSimOverviewTestHelper { + + public static BSimOverviewModel getOverviewModel(BSimOverviewProvider overviewProvider) { + return overviewProvider.getModel(); + } + + public static void setBSimOVerviewTestServer(BSimSearchPlugin plugin, + BSimOverviewDialog dialog, FunctionDatabase database) { + BSimServerInfo serverInfo = new TestBSimServerInfo(database); + Swing.runNow(() -> { + TestSFQueryServiceFactory factory = new TestSFQueryServiceFactory(database); + BSimSearchPluginTestHelper.setQueryServiceFactory(plugin, factory); + BSimServerManager serverManager = BSimSearchPluginTestHelper.getServerManager(plugin); + serverManager.addServer(serverInfo); + }); + Swing.runNow(() -> { + BSimSearchDialogTestHelper.setSelectedServer(dialog, serverInfo); + }); + } +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/dialog/BSimFilterPanelTest.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/dialog/BSimFilterPanelTest.java new file mode 100755 index 0000000000..98cf419d50 --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/dialog/BSimFilterPanelTest.java @@ -0,0 +1,351 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import static org.junit.Assert.*; + +import java.sql.SQLException; +import java.util.*; +import java.util.regex.Matcher; +import java.util.regex.Pattern; + +import org.junit.*; + +import ghidra.features.bsim.gui.AbstractBSimPluginTest; +import ghidra.features.bsim.gui.filters.*; +import ghidra.features.bsim.query.SQLFunctionDatabase; +import ghidra.features.bsim.query.client.*; +import ghidra.features.bsim.query.facade.FunctionDatabaseTestDouble; +import ghidra.features.bsim.query.protocol.BSimFilter; +import ghidra.program.database.symbol.FunctionSymbol; + +/** + * Tests the filtering components of BSim accessible from the UI. This will cover the + * following: + * + * - loading default filters + * - adding/removing/changing filters + * - input validation + * - proper construction of queries + * + */ +public class BSimFilterPanelTest extends AbstractBSimPluginTest { + + private Set selectedFunctions = new HashSet<>(); + private BSimFilterPanel filterPanel; + + @Before + public void setUp() throws Exception { + super.setUp(); + goTo(FUN1_ADDR); + performAction(searchAction, false); + + searchDialog = waitForDialogComponent(BSimSearchDialog.class); + filterPanel = BSimSearchDialogTestHelper.getFilterPanel(searchDialog); + } + + @After + public void tearDown() throws Exception { + close(searchDialog); + env.dispose(); + } + + @Test + public void testOneFilterInPanelByDefault() { + List widgets = filterPanel.getFilterWidgets(); + assertEquals(1, widgets.size()); + FilterWidget widget = widgets.get(0); + List filterTypes = widget.getChoosableFilterTypes(); + List baseFilters = BSimFilterType.getBaseFilters(); + + // should have all the base filters plus 2 for date earler, date later and 3 for + // function tags "KNOWN_LIBRARY", "HAS_UNIMPLEMENTED", and "HAS_BAD_DATA" + assertTrue(filterTypes.containsAll(baseFilters)); + assertEquals(baseFilters.size() + 5, filterTypes.size()); + } + + @Test + public void testAddFilter() { + List widgets = filterPanel.getFilterWidgets(); + assertEquals(1, widgets.size()); + + pressButtonByName(filterPanel, "Add Filter"); + + widgets = filterPanel.getFilterWidgets(); + assertEquals(2, widgets.size()); + + pressButtonByName(filterPanel, "Add Filter"); + + widgets = filterPanel.getFilterWidgets(); + assertEquals(3, widgets.size()); + + } + + @Test + public void testRemoveFilter() { + List widgets = filterPanel.getFilterWidgets(); + + pressButtonByName(filterPanel, "Add Filter"); + + widgets = filterPanel.getFilterWidgets(); + assertEquals(2, widgets.size()); + + pressButtonByName(widgets.get(0), "Delete Filter"); + widgets = filterPanel.getFilterWidgets(); + assertEquals(1, widgets.size()); + } + + @Test + public void testRemovingLastFilterLeavesItButIsBackToBlankFilter() { + FilterWidget widget = filterPanel.getFilterWidgets().get(0); + + assertEquals(new BlankBSimFilterType(), widget.getSelectedFilter()); + ExecutableNameBSimFilterType exeFilter = new ExecutableNameBSimFilterType(); + setFilter(widget, exeFilter, "bob"); + + assertEquals(exeFilter, widget.getSelectedFilter()); + assertEquals("bob", widget.getValues().get(0)); + + pressButtonByName(widget, "Delete Filter"); + + widget = filterPanel.getFilterWidgets().get(0); + assertEquals(new BlankBSimFilterType(), widget.getSelectedFilter()); + + } + + @Test + public void testFilterValidation_MD5() { + FilterWidget widget = filterPanel.getFilterWidgets().get(0); + setFilter(widget, new Md5BSimFilterType(), "123"); + assertFalse(widget.hasValidValue()); + + setFilter(widget, new Md5BSimFilterType(), "0123456789ABCDEF0123456789ABCEDF"); + assertTrue(hasValidValue(widget)); + } + + @Test + public void testFilterValidation_Dates() { + FilterWidget widget = filterPanel.getFilterWidgets().get(0); + DateEarlierBSimFilterType dateFilter = new DateEarlierBSimFilterType("Ingest Date"); + setFilter(widget, dateFilter, "123"); + assertFalse(widget.hasValidValue()); + + setFilter(widget, dateFilter, "09211974"); + assertFalse(hasValidValue(widget)); + + setFilter(widget, dateFilter, "January 4th, 2006"); + assertFalse(hasValidValue(widget)); + + setFilter(widget, dateFilter, "2001/07/11"); + assertTrue(hasValidValue(widget)); + + setFilter(widget, dateFilter, "09/21/1974"); + assertTrue(hasValidValue(widget)); + + setFilter(widget, dateFilter, "2001-01-01"); + assertTrue(hasValidValue(widget)); + } + + /** + * Tests that duplicate filters are correctly combined in the final + * query. + * + * ie: (compiler_name = 0) OR (compiler_name = 1) + * @throws SQLException if there's a problem creating the query + */ + @Test + public void testDuplicateFilters() throws SQLException { + + pressButtonByName(filterPanel, "Add Filter"); + List widgets = filterPanel.getFilterWidgets(); + + setFilter(widgets.get(0), new CompilerBSimFilterType(), "gcc"); + setFilter(widgets.get(1), new CompilerBSimFilterType(), "gcc2"); + + BSimSqlClause clause = getSqlClause(); + // Verify that we have two compiler clauses. + String clause1 = "name_compiler=1"; + String clause2 = "name_compiler=0"; + assertTrue(clause.whereClause().contains(clause1)); + assertTrue(clause.whereClause().contains(clause2)); + + // And verify that those two clauses are separated by an OR. + String glue = getTextBetween(clause.whereClause(), clause1, clause2); + assertTrue(glue.contains("OR")); + assertTrue(!glue.contains("AND")); + } + + /** + * Tests that multiple entries in a single filter are correctly parsed. + * + * @throws SQLException if there's a problem creating the query + */ + @Test + public void testCSVEntry() throws SQLException { + List widgets = filterPanel.getFilterWidgets(); + + setFilter(widgets.get(0), new CompilerBSimFilterType(), List.of("gcc", "gcc2", "gcc3")); + + BSimSqlClause clause = getSqlClause(); + + // Verify that we have three clauses. + String clause1 = "name_compiler=0"; + String clause2 = "name_compiler=1"; + String clause3 = "name_compiler=2"; + assertTrue(clause.whereClause().contains(clause1)); + assertTrue(clause.whereClause().contains(clause2)); + assertTrue(clause.whereClause().contains(clause3)); + + // And verify that those two clauses are separated by an OR. + String glue = getTextBetween(clause.whereClause(), clause1, clause2); + assertTrue(glue.contains("OR")); + assertTrue(!glue.contains("AND")); + + } + + /** + * Tests that multiple negative filters are correctly combined. + * + * ie: (name != bob) && (name != john) + * + * @throws SQLException if there's a problem creating the query + * + */ + @Test + public void testCombiningNegativeFilters() throws SQLException { + pressButtonByName(filterPanel, "Add Filter"); + List widgets = filterPanel.getFilterWidgets(); + + setFilter(widgets.get(0), new NotExecutableNameBSimFilterType(), "exename"); + setFilter(widgets.get(1), new NotExecutableNameBSimFilterType(), "othername"); + + BSimSqlClause clause = getSqlClause(); + // Verify that we have two compiler clauses. + String clause1 = "name_exec != 'exename'"; + String clause2 = "name_exec != 'othername'"; + assertTrue(clause.whereClause().contains(clause1)); + assertTrue(clause.whereClause().contains(clause2)); + + // And verify that those two clauses are separated by an OR. + String glue = getTextBetween(clause.whereClause(), clause1, clause2); + assertTrue(!glue.contains("OR")); + assertTrue(glue.contains("AND")); + } + + /** + * Tests that multiple positive filters are correctly combined. + * + * ie: (name == bob) || (name == john) + * + * @throws SQLException if there's a problem creating the query + */ + @Test + public void testCombiningPositiveFilters() throws SQLException { + pressButtonByName(filterPanel, "Add Filter"); + List widgets = filterPanel.getFilterWidgets(); + + setFilter(widgets.get(0), new CompilerBSimFilterType(), "gcc"); + setFilter(widgets.get(1), new ArchitectureBSimFilterType(), "x86:LE:64:default"); + + BSimSqlClause clause = getSqlClause(); + // Verify that we have two compiler clauses. + String clause1 = "name_compiler=0"; + String clause2 = "architecture=1"; + assertTrue(clause.whereClause().contains(clause1)); + assertTrue(clause.whereClause().contains(clause2)); + + // And verify that those two clauses are separated by an OR. + String glue = getTextBetween(clause.whereClause(), clause1, clause2); + assertTrue(!glue.contains("OR")); + assertTrue(glue.contains("AND")); + } + + /** + * Generates a fake set of resolution IDs to be used in generating filter + * queries. For the purpose of this test suite, the value of the IDs isn't + * important; we just have to have valid objects to pass to the query. + * + * @param exeFilter the BSim filter object + * @return the array of resolution IDs + */ + private IDSQLResolution[] createMockResolutionIDs(BSimFilter exeFilter) { + + IDSQLResolution[] idres = new IDSQLResolution[exeFilter.numAtoms()]; + for (int i = 0; i < idres.length; i++) { + idres[i] = new IDSQLResolution.Compiler("something"); + idres[i].id1 = i; + } + + return idres; + } + + /** + * Uses regex to search a given string for all text between two substrings of + * that full string. The position of the two substrings relative to eachother + * is unimportant. ie: str1 + + str2 will work just as well as + * str2 + + str1. This method will figure out what the correct + * order is. + * + * Note: This is used in this test suite to find out if two sql clauses are + * combined using AND's or OR's. + * + * @param fullString the full string containing str1 and str2 + * @param str1 one side of the search + * @param str2 the other side of the search + * @return all text between str1 and str2 + */ + private String getTextBetween(String fullString, String str1, String str2) { + + // First figure out which substring is on the left and which is on the right. + int index1 = fullString.indexOf(str1); + int index2 = fullString.indexOf(str2); + if (index1 > index2) { + String temp = str1; + str1 = str2; + str2 = temp; + } + + String regex = Pattern.quote(str1) + "(.*?)" + Pattern.quote(str2); + Pattern pattern = Pattern.compile(regex); + Matcher matcher = pattern.matcher(fullString); + while (matcher.find()) { + return matcher.group(1); + } + + return ""; + } + + private void setFilter(FilterWidget widget, BSimFilterType filter, String value) { + setFilter(widget, filter, List.of(value)); + } + + private void setFilter(FilterWidget widget, BSimFilterType filter, List values) { + runSwing(() -> widget.setFilter(filter, values)); + } + + private boolean hasValidValue(FilterWidget widget) { + return runSwing(() -> widget.hasValidValue()); + } + + private BSimSqlClause getSqlClause() throws SQLException { + BSimFilter bSimFilter = runSwing(() -> filterPanel.getFilterSet().getBSimFilter()); + IDSQLResolution[] resolutionIds = createMockResolutionIDs(bSimFilter); + SQLFunctionDatabase database = new FunctionDatabaseTestDouble(); + BSimSqlClause clause = SQLEffects.createFilter(bSimFilter, resolutionIds, database); + return clause; + } + +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/dialog/BSimSearchDialogTestHelper.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/dialog/BSimSearchDialogTestHelper.java new file mode 100644 index 0000000000..4792197296 --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/dialog/BSimSearchDialogTestHelper.java @@ -0,0 +1,59 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.dialog; + +import java.util.Set; + +import ghidra.features.bsim.gui.BSimSearchPlugin; +import ghidra.features.bsim.gui.BSimSearchPluginTestHelper; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.facade.TestBSimServerInfo; +import ghidra.features.bsim.query.facade.TestSFQueryServiceFactory; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.util.Swing; + +public class BSimSearchDialogTestHelper { + public static Set getSelectedFunctions(BSimSearchDialog dialog) { + return dialog.getSelectedFunction(); + } + + public static BSimFilterPanel getFilterPanel(BSimSearchDialog dialog) { + return dialog.getFilterPanel(); + } + + public static BSimServerInfo getSelectedServer(BSimSearchDialog dialog) { + return dialog.getServer(); + } + + public static void setSelectedServer(AbstractBSimSearchDialog dialog, BSimServerInfo server) { + dialog.setServer(server); + } + + public static void setBSimSearchTestServer(BSimSearchPlugin plugin, + BSimSearchDialog dialog, FunctionDatabase database) { + BSimServerInfo serverInfo = new TestBSimServerInfo(database); + Swing.runNow(() -> { + TestSFQueryServiceFactory factory = new TestSFQueryServiceFactory(database); + BSimSearchPluginTestHelper.setQueryServiceFactory(plugin, factory); + BSimServerManager serverManager = BSimSearchPluginTestHelper.getServerManager(plugin); + serverManager.addServer(serverInfo); + }); + Swing.runNow(() -> { + BSimSearchDialogTestHelper.setSelectedServer(dialog, serverInfo); + }); + } +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsTestHelper.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsTestHelper.java new file mode 100644 index 0000000000..615c4c8d7a --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/features/bsim/gui/search/results/BSimSearchResultsTestHelper.java @@ -0,0 +1,28 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.gui.search.results; + +public class BSimSearchResultsTestHelper { + public static BSimMatchResultsModel getSearchResultsModel(BSimSearchResultsProvider provider) { + return provider.getMatchesModel(); + } + + public static BSimExecutablesSummaryModel getExecutablesModel( + BSimSearchResultsProvider provider) { + return provider.getExecutablesModel(); + } + +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/inmemory/BSimH2DatabaseManagerTest.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/inmemory/BSimH2DatabaseManagerTest.java new file mode 100644 index 0000000000..3878d4caa6 --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/inmemory/BSimH2DatabaseManagerTest.java @@ -0,0 +1,304 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.query.inmemory; + +import static org.junit.Assert.*; + +import java.io.File; +import java.util.*; + +import org.junit.*; + +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.BSimServerInfo.DBType; +import ghidra.features.bsim.query.FunctionDatabase.Error; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager; +import ghidra.features.bsim.query.file.BSimH2FileDBConnectionManager.BSimH2FileDataSource; +import ghidra.features.bsim.query.protocol.CreateDatabase; +import ghidra.features.bsim.query.protocol.ResponseInfo; +import ghidra.framework.Application; +import ghidra.test.AbstractGhidraHeadedIntegrationTest; +import ghidra.util.Msg; +import utilities.util.FileUtilities; + +public class BSimH2DatabaseManagerTest extends AbstractGhidraHeadedIntegrationTest { + public static final String MEDIUM_NOSIZE = "medium_nosize"; + + //private static final String XML_SOURCE_DIR = System.getProperty("user.home") + "/sigs/postgres_test"; + + //private static final String TEST_DIR_NO_XML = System.getProperty("user.home") + "/sigs/empty"; + + @Before + public void setUp() { + cleanup(); + getTempDbDir().mkdir(); + } + + @After + public void tearDown() { + //cleanup(); + } + + private File getTempDbDir() { + return new File(Application.getUserTempDirectory(), "BSimH2Test"); + } + + private void cleanup() { + for (BSimH2FileDataSource ds : BSimH2FileDBConnectionManager.getAllDataSources()) { + ds.delete(); + } + FileUtilities.deleteDir(getTempDbDir()); + } + + private String getDbName(String name) { + return (new File(getTempDbDir(), name)).getAbsolutePath(); + } + + private BSimServerInfo getBsimServerInfo(String name) { + return new BSimServerInfo(DBType.file, null, -1, getDbName(name)); + } + + private BSimServerInfo createDatabase(String databaseName) { + return createDatabase(databaseName, null, null, null); + } + + private BSimServerInfo createDatabase(String databaseName, List tags, + List execats, String expectedError) { + + BSimServerInfo h2DbInfo = getBsimServerInfo(databaseName); + Msg.debug(this, "Creating H2 File DB: " + h2DbInfo); + + try (FunctionDatabase h2Database = BSimClientFactory.buildClient(h2DbInfo, false)) { + BSimH2FileDataSource bds = + BSimH2FileDBConnectionManager.getDataSource(h2Database.getServerInfo()); + assertEquals("Expected no connections", 0, bds.getActiveConnections()); + assertFalse(bds.exists()); + + CreateDatabase command = new CreateDatabase(); + command.info = new DatabaseInformation(); + // Put in fields provided on the command line + // If they are null, the template will fill them in + command.info.databasename = databaseName; // TODO: Unclear how this relates to full dbPath + command.config_template = MEDIUM_NOSIZE; + command.info.trackcallgraph = true; + if (tags != null) { + command.info.functionTags = tags; + } + if (execats != null) { + command.info.execats = execats; + } + ResponseInfo response = command.execute(h2Database); + if (response == null) { + if (expectedError != null) { + Error lastError = h2Database.getLastError(); + assertNotNull(lastError); + assertTrue(lastError.message.contains(expectedError)); + } + else { + fail("Create failed: " + h2Database.getLastError().message); + } + } + else { + assertNull(h2Database.getLastError()); + } + } + return h2DbInfo; + } + + @Test + public void testCreateDatabase() { + + BSimServerInfo h2DbInfo = getBsimServerInfo("test"); + BSimH2FileDataSource ds = BSimH2FileDBConnectionManager.getDataSourceIfExists(h2DbInfo); + assertNull(ds); + ds = BSimH2FileDBConnectionManager.getDataSource(h2DbInfo); + assertFalse(ds.exists()); + + createDatabase("test"); + + assertTrue(ds.exists()); + } + + @Test + public void testListingDatabases() { + + List dbList = new ArrayList(); + for (int i = 1; i <= 3; i++) { + // Create data source without creating database + BSimServerInfo h2DbInfo = getBsimServerInfo("test" + i); + BSimH2FileDataSource bds = BSimH2FileDBConnectionManager.getDataSource(h2DbInfo); + dbList.add(h2DbInfo); + assertFalse(bds.exists()); + } + + List actualDbList = new ArrayList(); + for (BSimH2FileDataSource bds : BSimH2FileDBConnectionManager.getAllDataSources()) { + actualDbList.add(bds.getServerInfo()); + } + Collections.sort(actualDbList); + + assertEquals(dbList, actualDbList); + } + + @Test + public void testDatabaseConfiguration() { + + List tags = new ArrayList<>(); + tags.add("tag1"); + tags.add("tag2"); + + List cats = new ArrayList<>(); + cats.add("cat1"); + cats.add("cat2"); + cats.add("cat3"); + + BSimServerInfo serverInfo = createDatabase("test1", tags, cats, null); + + try (FunctionDatabase fdb = serverInfo.getFunctionDatabase(false)) { + assertTrue(fdb.initialize()); + DatabaseInformation info = fdb.getInfo(); + assertEquals(2, info.functionTags.size()); + assertTrue(info.functionTags.contains("tag1")); + assertTrue(info.functionTags.contains("tag2")); + assertEquals(3, info.execats.size()); + assertTrue(info.execats.contains("cat1")); + assertTrue(info.execats.contains("cat2")); + assertTrue(info.execats.contains("cat3")); + } + } + + @Test + public void testCreateClientForNonExistentDB() { + + BSimServerInfo serverInfo = getBsimServerInfo("test"); + try (FunctionDatabase fdb = serverInfo.getFunctionDatabase(false)) { + assertFalse(fdb.initialize()); + Error lastError = fdb.getLastError(); + assertNotNull(lastError); + assertTrue(lastError.message.startsWith("Database does not exist: ")); + } + } + +// +// @Test +// public void testCreateDBandConnectClient() { +// List exeCats = new ArrayList<>(); +// exeCats.add("cat1"); +// exeCats.add("cat2"); +// List funcTags = new ArrayList<>(); +// funcTags.add("tag1"); +// funcTags.add("tag2"); +// funcTags.add("tag3"); +// BSimH2DatabaseManager.createDatabase("test", new File(XML_SOURCE_DIR), MEDIUM_NOSIZE, +// funcTags, exeCats); +// URL url = null; +// H2FunctionDatabase db = null; +// try { +// url = BSimClientFactory.deriveBSimURL(Handler.BSIM_IM_MEM_PROTOCOL + "://test"); +// db = (H2FunctionDatabase) BSimClientFactory.buildClient(url, false); +// if (!db.initialize()) { +// fail(); +// } +// } +// catch (MalformedURLException e) { +// fail(); +// } +// QueryInfo queryInfo = new QueryInfo(); +// ResponseInfo responseInfo = (ResponseInfo) db.query(queryInfo); +// assertEquals("test", responseInfo.info.databasename); +// +// List dbCats = responseInfo.info.execats; +// assertEquals(2, dbCats.size()); +// assertEquals("cat1", dbCats.get(0)); +// assertEquals("cat2", dbCats.get(1)); +// +// List dbTags = responseInfo.info.functionTags; +// assertEquals(3, dbTags.size()); +// assertEquals("tag1", dbTags.get(0)); +// assertEquals("tag2", dbTags.get(1)); +// assertEquals("tag3", dbTags.get(2)); +// +// assertTrue(responseInfo.info.trackcallgraph); +// +// QueryExeCount queryCount = new QueryExeCount(); +// queryCount.includeFakes = false; +// ResponseExe h2CountResponse = (ResponseExe) db.query(queryCount); +// File xmlSourceDir = new File(XML_SOURCE_DIR); +// File[] xmlFiles = xmlSourceDir.listFiles((d, n) -> n.startsWith("sigs_")); +// assertEquals(xmlFiles.length, h2CountResponse.recordCount); +// +// db.close(); +// assertTrue(BSimH2DatabaseManager.exists("test")); +// BSimH2DatabaseManager.closeDatabase("test"); +// assertFalse(BSimH2DatabaseManager.exists("test")); +// } +// +// @Test(expected = IllegalArgumentException.class) +// public void testBadFunctionTag() { +// List funcTags = new ArrayList<>(); +// funcTags.add("tag%1"); +// BSimH2DatabaseManager.createDatabase("test", new File(XML_SOURCE_DIR), MEDIUM_NOSIZE, +// funcTags, Collections.emptyList()); +// } +// +// @Test(expected = IllegalArgumentException.class) +// public void testBadExecutableCategory() { +// List exeCats = new ArrayList<>(); +// exeCats.add("cat%1"); +// BSimH2DatabaseManager.createDatabase("test", new File(XML_SOURCE_DIR), MEDIUM_NOSIZE, +// Collections.emptyList(), exeCats); +// } +// +// @Test(expected = IllegalArgumentException.class) +// public void testDuplicateFunctionTags() { +// List funcTags = new ArrayList<>(); +// funcTags.add("tag1"); +// funcTags.add("tag1"); +// BSimH2DatabaseManager.createDatabase("test", new File(XML_SOURCE_DIR), MEDIUM_NOSIZE, +// funcTags, Collections.emptyList()); +// } +// +// @Test(expected = IllegalArgumentException.class) +// public void testDuplicateExecutableCategories() { +// List exeCats = new ArrayList<>(); +// exeCats.add("cat1"); +// exeCats.add("cat1"); +// BSimH2DatabaseManager.createDatabase("test", new File(XML_SOURCE_DIR), MEDIUM_NOSIZE, +// Collections.emptyList(), exeCats); +// } +// +// @Test +// public void testPing() { +// String name = "ping_test"; +// assertFalse(BSimH2DatabaseManager.exists(name)); +// BSimH2DatabaseManager.createDatabase(name, new File(XML_SOURCE_DIR), MEDIUM_NOSIZE, +// Collections.emptyList(), Collections.emptyList()); +// assertTrue(BSimH2DatabaseManager.exists(name)); +// BSimH2DatabaseManager.closeDatabase(name); +// assertFalse(BSimH2DatabaseManager.exists(name)); +// } +// +// @Test +// public void testNoSigFilesInDir() { +// String name = "test"; +// assertFalse(BSimH2DatabaseManager.exists(name)); +// BSimH2DatabaseManager.createDatabase(name, new File(TEST_DIR_NO_XML), MEDIUM_NOSIZE, +// Collections.emptyList(), Collections.emptyList()); +// assertFalse(BSimH2DatabaseManager.exists(name)); +// } + +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/test/BSimServerTest.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/test/BSimServerTest.java new file mode 100755 index 0000000000..b1073152bf --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/test/BSimServerTest.java @@ -0,0 +1,732 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.query.test; + +import static org.junit.Assert.*; + +import java.io.*; +import java.net.URL; +import java.sql.Date; +import java.time.Instant; +import java.util.*; + +import org.junit.*; +import org.xml.sax.SAXException; + +import generic.jar.ResourceFile; +import generic.lsh.vector.*; +import generic.util.Path; +import ghidra.GhidraTestApplicationLayout; +import ghidra.app.util.headless.HeadlessAnalyzer; +import ghidra.app.util.headless.HeadlessOptions; +import ghidra.features.bsim.gui.filters.ExecutableCategoryBSimFilterType; +import ghidra.features.bsim.gui.filters.HasNamedChildBSimFilterType; +import ghidra.features.bsim.query.*; +import ghidra.features.bsim.query.FunctionDatabase.Error; +import ghidra.features.bsim.query.client.tables.ExeTable.ExeTableOrderColumn; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.ingest.BSimLaunchable; +import ghidra.features.bsim.query.protocol.*; +import ghidra.framework.*; +import ghidra.framework.client.HeadlessClientAuthenticator; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +// These tests require specific data files and paths to be set up. See BSimServerTestUtil. +// The "ignore" directive is to prevent these from running as part of the automated nightly tests. +@Ignore +public class BSimServerTest { + private static final String PROPERTIES_FILE = "RegressionSignatures.properties"; + + private static BSimServerTestUtil util; + private static LSHVectorFactory vectorFactory; + private static FunctionDatabase client; + private static BSimLaunchable bulk; + private static ResourceFile dumpFile; + private static DescriptionManager originalBash; + + private static XmlPullParser getParser(ResourceFile file) { + XmlPullParser parser; + try { + InputStream input = file.getInputStream(); + parser = new NonThreadedXmlPullParserImpl(input, "BSim test parser", + SpecXmlUtils.getXmlHandler(), false); + } + catch (SAXException e) { + return null; + } + catch (IOException e) { + return null; + } + return parser; + } + + @BeforeClass + public static void setUp() throws Exception { + util = new BSimServerTestUtil(); + util.verifyDirectories(); + GhidraTestApplicationLayout layout = + new GhidraTestApplicationLayout(new File(util.ghidraDir)); + ApplicationConfiguration config = new HeadlessGhidraApplicationConfiguration(); + Application.initializeApplication(layout, config); + ghidra.framework.protocol.ghidra.Handler.registerHandler(); /// Register ghidra: protocol + ghidra.features.bsim.query.postgresql.Handler.registerHandler(); // Register postgresql: protocol + HeadlessClientAuthenticator.installHeadlessClientAuthenticator(null, null, true); + bulk = new BSimLaunchable(); + + util.verifyRaw(); + File propFile = new File(util.xmlDir, PROPERTIES_FILE); + if (!propFile.isFile()) { + createPropertiesFile(); + runHeadless(); + } + + util.startServer(); + doIngest(); + BSimServerInfo bsimServerInfo; + try { + bsimServerInfo = new BSimServerInfo(new URL(util.bsimURLString)); + } + catch (Exception e) { + throw new AssertionError(e); + } + client = BSimClientFactory.buildClient(bsimServerInfo, false); + if (!client.initialize()) { + throw new IOException("Unable to connect to server"); + } + vectorFactory = client.getLSHVectorFactory(); + + ResourceFile xmlFile = + new ResourceFile(new ResourceFile(util.xmlDir), "sigs_" + BSimServerTestUtil.BASH_MD5); + if (!xmlFile.isFile()) { + throw new IOException("Basic signature generation did not happen"); + } + XmlPullParser parser = getParser(xmlFile); + originalBash = new DescriptionManager(); + originalBash.restoreXml(parser, vectorFactory); + parser.dispose(); + } + + @AfterClass + public static void shutdown() throws Exception { + if (client != null) { + client.close(); + } + util.shutdownServer(); + if (dumpFile != null && dumpFile.exists()) { + dumpFile.delete(); + } + } + + @Test + public void testLibHistoryXml() { + ResourceFile xmlFile = new ResourceFile(new ResourceFile(util.xmlDir), + "sigs_" + BSimServerTestUtil.LIBHISTORY_MD5); + assertTrue(xmlFile.isFile()); + XmlPullParser parser = getParser(xmlFile); + DescriptionManager manager = new DescriptionManager(); + try { + manager.restoreXml(parser, vectorFactory); + assertTrue(manager.getExecutableRecordSet().size() == 2); + ExecutableRecord eRec = manager.findExecutable(BSimServerTestUtil.LIBHISTORY_MD5); + // make sure basic meta-data comes through + assertTrue(eRec.getNameExec().equals("libhistory.so.7.0")); + assertTrue(eRec.getNameCompiler().equals("gcc")); + assertTrue(eRec.getPath().equals("raw")); + assertTrue(eRec.getRepository().equals("ghidra://localhost/repo")); + assertTrue(eRec.hasCategory("Test Category", "shared")); + ExecutableRecord libRec = manager.findExecutable("unknown", "x86:LE:64:default", ""); + assertTrue(libRec.isLibrary()); + FunctionDescription fDesc = manager.findFunctionByName("close", libRec); + assertNotNull(fDesc); + assertEquals(fDesc.getAddress(), -1); + fDesc = manager.findFunctionByName("read_history_range", eRec); + assertNotNull(fDesc); + assertEquals(fDesc.getAddress(), 0x105f60); + FunctionDescription addHistory = null; + FunctionDescription malloc = null; + for (CallgraphEntry entry : fDesc.getCallgraphRecord()) { + String name = entry.getFunctionDescription().getFunctionName(); + if (name.equals("add_history")) { + addHistory = entry.getFunctionDescription(); + } + else if (name.equals("malloc")) { + malloc = entry.getFunctionDescription(); + } + } + assertNotNull(addHistory); + assertNotNull(malloc); + assertEquals(addHistory.getAddress(), 0x102770); + assertEquals(malloc.getAddress(), -1); + assertTrue(addHistory.getExecutableRecord() == eRec); // Should be same object + assertTrue(malloc.getExecutableRecord() == libRec); + } + catch (LSHException e) { + Assert.fail("Failure processing libhistory"); + } + } + + @Test + public void testBashLibReadline() { + try { + ResourceFile xmlFile = new ResourceFile(new ResourceFile(util.xmlDir), + "sigs_" + BSimServerTestUtil.LIBREADLINE_MD5); + XmlPullParser parser = getParser(xmlFile); + DescriptionManager manager = new DescriptionManager(); + manager.restoreXml(parser, vectorFactory); + parser.dispose(); + assertEquals(manager.getExecutableRecordSet().size(), 2); + ExecutableRecord bashRec = originalBash.findExecutable(BSimServerTestUtil.BASH_MD5); + assertTrue(bashRec.hasCategory("Test Category", "static")); + ExecutableRecord readRec = manager.findExecutable(BSimServerTestUtil.LIBREADLINE_MD5); + assertTrue(readRec.hasCategory("Test Category", "shared")); + // Comparing function "history_filename" + FunctionDescription bashFunc = + originalBash.findFunction("FUN_001cc840", 0x1cc840, bashRec); + FunctionDescription readFunc = manager.findFunction("FUN_00134a70", 0x134a70, readRec); + VectorCompare compareData = new VectorCompare(); + double sig = bashFunc.getSignatureRecord() + .getLSHVector() + .compare(readFunc.getSignatureRecord().getLSHVector(), compareData); + assertTrue(sig > 0.99999); + } + catch (LSHException e) { + Assert.fail("Failure processing bash and libreadline"); + } + } + + private static void compareExe(DescriptionManager manager1, DescriptionManager manager2, + String md5) throws Exception { + ExecutableRecord eRec1 = manager1.findExecutable(md5); + ExecutableRecord eRec2 = manager2.findExecutable(md5); + Iterator iter = manager1.listFunctions(eRec1); + while (iter.hasNext()) { + FunctionDescription func1 = iter.next(); + FunctionDescription func2 = + manager2.findFunction(func1.getFunctionName(), func1.getAddress(), eRec2); + assertEquals(func1.getFlags(), func2.getFlags()); + if (func1.getSignatureRecord() == null) { + assertTrue(func2.getSignatureRecord() == null); + } + else { + assertNotNull(func2.getSignatureRecord()); + assertTrue(func1.getSignatureRecord() + .getLSHVector() + .equals(func2.getSignatureRecord().getLSHVector())); + } + if (func1.getCallgraphRecord() == null) { + assertTrue(func2.getCallgraphRecord() == null); + continue; + } + assertNotNull(func2.getCallgraphRecord()); + func1.sortCallgraph(); + func2.sortCallgraph(); + Iterator iter1 = func1.getCallgraphRecord().iterator(); + Iterator iter2 = func2.getCallgraphRecord().iterator(); + while (iter1.hasNext()) { + assertTrue(iter2.hasNext()); + FunctionDescription call1 = iter1.next().getFunctionDescription(); + FunctionDescription call2 = iter2.next().getFunctionDescription(); + assertTrue(call1.equals(call2)); + } + } + } + + @Test + public void testDumpFile() { + try { + assertNotNull(dumpFile); + assertTrue(dumpFile.exists()); + XmlPullParser parser = getParser(dumpFile); + DescriptionManager manager1 = new DescriptionManager(); + manager1.restoreXml(parser, vectorFactory); + parser.dispose(); + compareExe(manager1, originalBash, BSimServerTestUtil.BASH_MD5); + } + catch (Exception e) { + Assert.fail("Failed to perform dumpexexml: " + e.getMessage()); + } + } + + private static void testForError(QueryResponseRecord response) throws LSHException { + if (response == null) { + Error lastError = client.getLastError(); + if (lastError == null) { + throw new LSHException("Unknown error"); + } + throw new LSHException(lastError.message); + } + } + + @Test + public void testQueryInfo() throws LSHException { + QueryInfo queryInfo = new QueryInfo(); + ResponseInfo response = queryInfo.execute(client); + testForError(response); + DatabaseInformation info = response.info; + assertTrue(info.databasename.equals("TestName")); + assertTrue(info.owner.equals("TestOwner")); + assertTrue(info.description.equals("TestDescription")); + assertEquals(info.settings, 0x49); + assertTrue(info.trackcallgraph); + assertNotNull(info.execats); + assertEquals(info.execats.size(), 1); + assertTrue(info.execats.get(0).equals("Test Category")); + } + + private void compareExecutableRecords(ExecutableRecord exe1, ExecutableRecord exe2) { + assertEquals(exe1.getMd5(), exe2.getMd5()); + assertEquals(exe1.getNameExec(), exe2.getNameExec()); + assertEquals(exe1.getArchitecture(), exe2.getArchitecture()); + assertEquals(exe1.getNameCompiler(), exe2.getNameCompiler()); + assertEquals(exe1.getPath(), exe2.getPath()); + } + + @Test + public void testQueryExeInfo() throws LSHException { + ExecutableRecord libHistory = new ExecutableRecord(BSimServerTestUtil.LIBHISTORY_MD5, + "libhistory.so.7.0", "gcc", "x86:LE:64:default", null, null, null, "raw"); + ExecutableRecord libReadline = new ExecutableRecord(BSimServerTestUtil.LIBREADLINE_MD5, + "libreadline.so.7.0", "gcc", "x86:LE:64:default", null, null, null, "raw"); + QueryExeInfo queryExeInfo = new QueryExeInfo(); + queryExeInfo.includeFakes = true; + ResponseExe responseExe = queryExeInfo.execute(client); + testForError(responseExe); + assertEquals(responseExe.recordCount, 3); + ExecutableRecord exe1 = responseExe.records.get(0); + compareExecutableRecords(exe1, libHistory); + assertEquals(exe1.getExeCategoryAlphabetic("Test Category"), "shared"); + ExecutableRecord exe2 = responseExe.records.get(1); + compareExecutableRecords(exe2, libReadline); + assertEquals(exe2.getExeCategoryAlphabetic("Test Category"), "shared"); + ExecutableRecord exe3 = responseExe.records.get(2); + assertEquals(exe3.getMd5(), "bbbbbbbbaaaaaaaa4b13cd7905584d9f"); + assertEquals(exe3.getNameExec(), "unknown"); + + queryExeInfo = new QueryExeInfo(); + queryExeInfo.filterMd5 = BSimServerTestUtil.LIBREADLINE_MD5; + responseExe = queryExeInfo.execute(client); + testForError(responseExe); + assertEquals(responseExe.recordCount, 1); + exe1 = responseExe.records.get(0); + compareExecutableRecords(exe1, libReadline); + + queryExeInfo = new QueryExeInfo(); + queryExeInfo.filterMd5 = "0a860"; // Partial md5 + responseExe = queryExeInfo.execute(client); + testForError(responseExe); + assertEquals(responseExe.recordCount, 1); + exe1 = responseExe.getDescriptionManager().findExecutable("libhistory.so.7.0", null, null); + compareExecutableRecords(exe1, libHistory); + + queryExeInfo = new QueryExeInfo(); + queryExeInfo.filterExeName = "lib"; + queryExeInfo.sortColumn = ExeTableOrderColumn.NAME; + responseExe = queryExeInfo.execute(client); + testForError(responseExe); + assertEquals(responseExe.records.size(), 2); + exe1 = responseExe.records.get(0); + exe2 = responseExe.records.get(1); + compareExecutableRecords(exe1, libHistory); + compareExecutableRecords(exe2, libReadline); + + QueryExeCount queryExeCount = new QueryExeCount(); + responseExe = queryExeCount.execute(client); + testForError(responseExe); + assertEquals(responseExe.recordCount, 2); + } + + @Test + public void testQueryName() throws LSHException { + QueryName queryName = new QueryName(); + queryName.spec.exename = "libhistory.so.7.0"; + queryName.funcname = "history_arg_extract"; + ResponseName responseName = queryName.execute(client); + testForError(responseName); + assertTrue(responseName.uniqueexecutable); + ExecutableRecord eRec = + responseName.manage.findExecutable(BSimServerTestUtil.LIBHISTORY_MD5); + assertTrue(eRec.getNameExec().equals("libhistory.so.7.0")); + assertTrue(eRec.getMd5().equals(BSimServerTestUtil.LIBHISTORY_MD5)); + assertTrue(eRec.getNameCompiler().equals("gcc")); + assertTrue(eRec.getPath().equals("raw")); + Iterator iter = responseName.manage.listAllFunctions(); + assertTrue(iter.hasNext()); + FunctionDescription func = iter.next(); + assertFalse(iter.hasNext()); // Should be exactly one function + assertTrue(func.getFunctionName().equals("history_arg_extract")); + assertEquals(func.getAddress(), 0x103d40); + SignatureRecord sigRec = func.getSignatureRecord(); + assertNotNull(sigRec); + assertEquals(sigRec.getCount(), 2); + ExecutableRecord bashRec = originalBash.findExecutable(BSimServerTestUtil.BASH_MD5); + FunctionDescription bashFunc = + originalBash.findFunctionByName("history_arg_extract", bashRec); + assertNotNull(bashFunc); + VectorCompare vectorCompare = new VectorCompare(); + double sim = sigRec.getLSHVector() + .compare(bashFunc.getSignatureRecord().getLSHVector(), vectorCompare); + assertTrue(sim > 0.8); + assertTrue(sim < 0.999); + } + + private static QueryResponseRecord doStagedQuery(BSimQuery query, + StagingManager stagingManager) throws LSHException { + + boolean haveMore = stagingManager.initialize(query); + query.buildResponseTemplate(); + + QueryResponseRecord globalResponse = query.getResponse(); + + while (haveMore) { + // Get the current staged form of the query + BSimQuery stagedQuery = stagingManager.getQuery(); + QueryResponseRecord response = stagedQuery.execute(client); + if (response != null) { + if (globalResponse != response) { + globalResponse.mergeResults(response); // Merge the staged response with the global response + } + + haveMore = stagingManager.nextStage(); + if (haveMore) { + stagedQuery.clearResponse(); // Make space for next stage + } + } + else { + throw new LSHException(client.getLastError().message); + } + } + + return globalResponse; + } + + private static void testSimilarityResult(SimilarityResult simRes) { + assertEquals(simRes.getTotalCount(), 2); + Iterator iter = simRes.iterator(); + SimilarityNote note1 = iter.next(); + SimilarityNote note2 = iter.next(); + FunctionDescription func1 = note1.getFunctionDescription(); + FunctionDescription func2 = note2.getFunctionDescription(); + assertTrue(func1.getExecutableRecord().getNameExec().equals("libhistory.so.7.0")); + assertTrue(func2.getExecutableRecord().getNameExec().equals("libreadline.so.7.0")); + assertNotNull(func1.getSignatureRecord()); + assertNotNull(func2.getSignatureRecord()); + assertNotNull(simRes.getBase().getSignatureRecord()); + LSHVector baseVector = simRes.getBase().getSignatureRecord().getLSHVector(); + VectorCompare vectorCompare = new VectorCompare(); + double sim1 = func1.getSignatureRecord().getLSHVector().compare(baseVector, vectorCompare); + assertEquals(note1.getSimilarity(), sim1, 0.0001); + double sim2 = func2.getSignatureRecord().getLSHVector().compare(baseVector, vectorCompare); + assertEquals(note2.getSimilarity(), sim2, 0.0001); + } + + @Test + public void testQueryNearest() throws LSHException { + QueryNearest queryNearest = new QueryNearest(); + ExecutableRecord bashRec = originalBash.findExecutable(BSimServerTestUtil.BASH_MD5); + FunctionDescription func1 = originalBash.findFunctionByName("_rl_adjust_point", bashRec); + FunctionDescription func2 = originalBash.findFunctionByName("_rl_compare_chars", bashRec); + FunctionDescription func3 = originalBash.findFunctionByName("add_history", bashRec); + FunctionDescription func4 = originalBash.findFunctionByName("get_history_event", bashRec); + queryNearest.manage.transferSettings(originalBash); + queryNearest.manage.transferFunction(func1, true); + queryNearest.manage.transferFunction(func2, true); + queryNearest.manage.transferFunction(func3, true); + queryNearest.manage.transferFunction(func4, true); + StagingManager functionStage = new FunctionStaging(2); + QueryResponseRecord response = doStagedQuery(queryNearest, functionStage); + testForError(response); + ResponseNearest respNearest = (ResponseNearest) response; + respNearest.sort(); + int matchCount = 0; + for (SimilarityResult simRes : respNearest.result) { + if (simRes.getBase().getAddress() == func1.getAddress()) { + assertTrue(func1.equals(simRes.getBase())); + matchCount += 1; + } + else if (simRes.getBase().getAddress() == func2.getAddress()) { + assertTrue(func2.equals(simRes.getBase())); + matchCount += 1; + } + else if (simRes.getBase().getAddress() == func3.getAddress()) { + assertTrue(func3.equals(simRes.getBase())); + matchCount += 1; + } + else if (simRes.getBase().getAddress() == func4.getAddress()) { + assertTrue(func4.equals(simRes.getBase())); + matchCount += 1; + } + testSimilarityResult(simRes); + } + assertEquals(matchCount, 4); // Make sure we hit all functions + } + + @Test + public void testQueryVector() throws LSHException { + QueryNearestVector queryVector = new QueryNearestVector(); + ExecutableRecord bashRec = originalBash.findExecutable(BSimServerTestUtil.BASH_MD5); + FunctionDescription func = originalBash.findFunctionByName("_rl_kill_kbd_macro", bashRec); + queryVector.manage.transferSettings(originalBash); + queryVector.manage.transferFunction(func, true); + ResponseNearestVector respVector = queryVector.execute(client); + testForError(respVector); + respVector.sort(); + assertEquals(respVector.totalvec, 1); + assertEquals(respVector.totalmatch, 2); + assertEquals(respVector.uniquematch, 0); // 1 vector matches 2 functions + Iterator iter = respVector.result.iterator(); + SimilarityVectorResult simVecRes = iter.next(); + assertFalse(iter.hasNext()); + assertTrue(simVecRes.getBase().equals(func)); + assertEquals(simVecRes.getTotalCount(), 2); + Iterator iter2 = simVecRes.iterator(); + VectorResult vec1 = iter2.next(); + VectorResult vec2 = iter2.next(); + assertFalse(iter2.hasNext()); + if (vec1.sim > vec2.sim) { + VectorResult tmp = vec1; + vec1 = vec2; + vec2 = tmp; + } + VectorCompare vectorCompare = new VectorCompare(); + LSHVector baseVector = func.getSignatureRecord().getLSHVector(); + double sim1 = baseVector.compare(vec1.vec, vectorCompare); + assertEquals(sim1, vec1.sim, 0.0001); + assertEquals(vec1.hitcount, 1); + double sim2 = baseVector.compare(vec2.vec, vectorCompare); + assertEquals(sim2, vec2.sim, 0.0001); + assertEquals(vec1.hitcount, 1); + assertTrue(vec2.sim > 0.999); // Second vector should be 1.0 match + } + + @Test + public void testChildFilter() throws LSHException { + QueryNearest query = new QueryNearest(); + query.manage.transferSettings(originalBash); + ExecutableRecord bashRec = originalBash.findExecutable(BSimServerTestUtil.BASH_MD5); + FunctionDescription funcDesc = originalBash.findFunctionByName("_rl_errmsg", bashRec); + query.manage.transferFunction(funcDesc, true); + query.bsimFilter = new BSimFilter(); + query.bsimFilter.addAtom(new HasNamedChildBSimFilterType(), "[unknown]__fprintf_chk"); + ResponseNearest respNearest = query.execute(client); + testForError(respNearest); + assertEquals(respNearest.totalfunc, 1); + assertEquals(respNearest.totalmatch, 1); // Filtered all but one response + assertEquals(respNearest.uniquematch, 1); + ExecutableRecord eRec = respNearest.manage.getExecutableRecordSet().first(); + assertTrue(eRec.getNameExec().equals("libreadline.so.7.0")); + assertTrue(eRec.getMd5().equals(BSimServerTestUtil.LIBREADLINE_MD5)); + assertEquals(respNearest.result.size(), 1); + SimilarityResult simRes = respNearest.result.get(0); + assertEquals(simRes.size(), 1); // Only one function returned + assertEquals(simRes.getTotalCount(), 3); // 3 functions similar to vector + SimilarityNote note = simRes.iterator().next(); + assertTrue(note.getSimilarity() > 0.800); + FunctionDescription resFunc = note.getFunctionDescription(); + assertTrue(resFunc.getFunctionName().equals("FUN_0011ece0")); + assertEquals(resFunc.getAddress(), 0x11ece0); + assertEquals(resFunc.getSignatureRecord().getCount(), 1); // only function with this exact vector + } + + @Test + public void testUpdate() throws Exception { + QueryUpdate update = new QueryUpdate(); + + Date dt = new Date(Instant.parse("2010-12-25T10:15:30.00Z").toEpochMilli()); + ExecutableRecord exerec = update.manage.newExecutableRecord( + BSimServerTestUtil.LIBREADLINE_MD5, "libreadline.so.7.0", "gcc", "x86:LE:64:default", + dt, "ghidra://localhost/repo", "/raw", null); + List catrec = new ArrayList<>(); + catrec.add(new CategoryRecord("Test Category", "SHARED!")); + update.manage.setExeCategories(exerec, catrec); + update.manage.newFunctionDescription("my_remove_history", 0x131c00, exerec); + ResponseUpdate respUpdate = update.execute(client); + testForError(respUpdate); + assertEquals(respUpdate.exeupdate, 1); + assertEquals(respUpdate.funcupdate, 1); + assertTrue(respUpdate.badexe.isEmpty()); + assertTrue(respUpdate.badfunc.isEmpty()); + if (util.isElasticSearch) { + Thread.sleep(2000); // Give chance for refresh timer to expire + } + QueryNearest nearest = new QueryNearest(); + nearest.manage.transferSettings(originalBash); + + ExecutableRecord bash = originalBash.findExecutable(BSimServerTestUtil.BASH_MD5); + FunctionDescription desc = originalBash.findFunctionByName("remove_history", bash); + nearest.manage.transferFunction(desc, true); + nearest.bsimFilter = new BSimFilter(); + nearest.bsimFilter.addAtom(new ExecutableCategoryBSimFilterType("Test Category"), + "SHARED!"); + ResponseNearest respNearest = nearest.execute(client); + testForError(respNearest); + assertEquals(respNearest.totalfunc, 1); + assertEquals(respNearest.totalmatch, 1); + assertEquals(respNearest.uniquematch, 1); + SimilarityResult simRes = respNearest.result.get(0); + assertTrue(simRes.getBase().equals(desc)); // base should match original function + assertEquals(simRes.size(), 1); + assertEquals(simRes.getTotalCount(), 2); // The filtered libhistory version is also counted here + SimilarityNote note = simRes.iterator().next(); + FunctionDescription resFunc = note.getFunctionDescription(); + assertTrue(resFunc.getFunctionName().equals("my_remove_history")); + ExecutableRecord resRec = resFunc.getExecutableRecord(); + assertTrue(resRec.getDate().equals(dt)); + + // Restore the original records + update = new QueryUpdate(); + exerec = update.manage.newExecutableRecord(BSimServerTestUtil.LIBREADLINE_MD5, + "libreadline.so.7.0", "gcc", "x86:LE:64:default", bash.getDate(), + "ghidra://localhost/repo", "/raw", null); + catrec = new ArrayList<>(); + catrec.add(new CategoryRecord("Test Category", "shared")); + update.manage.setExeCategories(exerec, catrec); + update.manage.newFunctionDescription("remove_history", 0x131c00, exerec); + testForError(update.execute(client)); + if (util.isElasticSearch) { + Thread.sleep(2000); // Give chance for refresh timer to expire + } + } + + private static void doIngest() throws Exception { + if (dumpFile != null && dumpFile.exists()) { + return; + } + runCreateDatabase(); + if (util.isElasticSearch) { + Thread.sleep(2000); // Give chance for refresh timer to expire + } + runSetMetaData(); + runSetExeCategory(); + runDropIndex(); + if (util.isElasticSearch) { + Thread.sleep(2000); // Give chance for refresh timer to expire + } + runCommitSigs(); + runRebuildIndex(); + if (util.isElasticSearch) { + Thread.sleep(2000); // Give chance for refresh timer to expire + } + runDumpFile(BSimServerTestUtil.BASH_MD5); + dumpFile = + new ResourceFile(new ResourceFile(util.testDir), "sigs_" + BSimServerTestUtil.BASH_MD5); + runDelete(); + if (util.isElasticSearch) { + Thread.sleep(2000); // Give chance for refresh timer to expire + } + } + + private static void createPropertiesFile() throws IOException { + File props = new File(util.xmlDir, PROPERTIES_FILE); + FileWriter writer = new FileWriter(props); + writer.write("RegressionSignatures: Working directory = " + util.xmlDir + '\n'); + writer.close(); + } + + private static void runHeadless() throws IOException { + HeadlessAnalyzer analyzer = HeadlessAnalyzer.getInstance(); + HeadlessOptions options = analyzer.getOptions(); + List preScripts = new ArrayList<>(); + List postScripts = new ArrayList<>(); + List inputFiles = new ArrayList<>(); + + options.setPropertiesFileDirectories(util.xmlDir); + Path scriptPath = new Path(Path.GHIDRA_HOME + "/Features/BSim/other/testscripts"); + options.setScriptDirectories(scriptPath.getPath().getAbsolutePath()); + + inputFiles.add(new File(util.rawDir)); + preScripts.add("TailoredAnalysis.java"); + preScripts.add("InstallMetadataTest.java"); + postScripts.add("RegressionSignatures.java"); + + options.setPreScripts(preScripts); + options.setPostScripts(postScripts); + + analyzer.processLocal(util.projectDir, util.repoName, "/", inputFiles); + } + + private static void runCreateDatabase() throws Exception { + String params[] = new String[3]; + params[0] = "createdatabase"; + params[1] = util.bsimURLString; + params[2] = "medium_64"; + bulk.run(params); + } + + private static void runSetMetaData() throws Exception { + String params[] = new String[5]; + params[0] = "setmetadata"; + params[1] = util.bsimURLString; + params[2] = "name=TestName"; + params[3] = "owner=TestOwner"; + params[4] = "description=TestDescription"; + bulk.run(params); + } + + private static void runSetExeCategory() throws Exception { + String params[] = new String[3]; + params[0] = "addexecategory"; + params[1] = util.bsimURLString; + params[2] = "Test Category"; + bulk.run(params); + } + + private static void runDropIndex() throws Exception { + if (util.isH2Database) { + return; + } + String params[] = new String[2]; + params[0] = "dropindex"; + params[1] = util.bsimURLString; + bulk.run(params); + } + + private static void runCommitSigs() throws Exception { + String params[] = new String[3]; + params[0] = "commitsigs"; + params[1] = util.bsimURLString; + params[2] = util.xmlDir; + bulk.run(params); + } + + private static void runRebuildIndex() throws Exception { + if (util.isH2Database) { + return; + } + String params[] = new String[2]; + params[0] = "rebuildindex"; + params[1] = util.bsimURLString; + bulk.run(params); + } + + private static void runDelete() throws Exception { + String params[] = new String[3]; + params[0] = "delete"; + params[1] = util.bsimURLString; + params[2] = "md5=" + BSimServerTestUtil.BASH_MD5; + bulk.run(params); + } + + private static void runDumpFile(String md5) throws Exception { + String params[] = new String[4]; + params[0] = "dumpsigs"; + params[1] = util.bsimURLString; + params[2] = util.testDir; + params[3] = "md5=" + BSimServerTestUtil.BASH_MD5; + bulk.run(params); + } +} diff --git a/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/test/BSimServerTestUtil.java b/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/test/BSimServerTestUtil.java new file mode 100755 index 0000000000..7a1e2f1728 --- /dev/null +++ b/Ghidra/Features/BSim/src/test.slow/java/ghidra/query/test/BSimServerTestUtil.java @@ -0,0 +1,192 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.query.test; + +import java.io.*; + +import ghidra.features.bsim.query.BSimControlLaunchable; +import ghidra.util.MD5Utilities; +import utilities.util.FileUtilities; + +/** + * The TEST_DIRECTORY String should be changed to point to a directory that will + * hold data for the server and for the tests. To start, this directory should contain + * a subdirectory "raw", and within this subdirectory should be the following 3 specific binary + * executables: + * libreadline.so.7.0 + * libhistory.so.7.0 + * bash + * + * all pulled from Ubuntu 18.04.5. + */ +public class BSimServerTestUtil { + private static final String HOST_URL = "postgresql://localhost"; +// private static final String HOST_URL = "https://localhost:9200"; +// private static final String HOST_URL = "file:///tmp/bsimtest/db"; + private static final String TEST_DIRECTORY = "/tmp/bsimtest"; + public static final String REPO_NAME = "repo"; + public static final String LIBHISTORY_MD5 = "0a860a716d5bec97c64db652549b72fd"; + public static final String LIBREADLINE_MD5 = "71b5761b43b840eb88d053790deaf77c"; + public static final String BASH_MD5 = "557c0271e30cf474e0f46f93721fd1ba"; + public String repoName; + public String bsimURLString = HOST_URL + '/' + REPO_NAME; + public String testDir; + public String ghidraDir; + public String projectDir; + public String rawDir; + public String xmlDir; + public String serverDir; + public String serverTouchDir; + public boolean isElasticSearch; + public boolean isH2Database; + + public BSimServerTestUtil() { + testDir = TEST_DIRECTORY; + ghidraDir = TEST_DIRECTORY + "/ghidra"; + repoName = REPO_NAME; + projectDir = testDir + "/project"; + rawDir = testDir + "/raw"; + xmlDir = testDir + "/xml"; + serverDir = testDir + "/db"; + serverTouchDir = testDir + "/servertouch"; + isElasticSearch = HOST_URL.startsWith("http") || HOST_URL.startsWith("elastic"); + isH2Database = HOST_URL.startsWith("file"); + } + + public void verifyDirectories() throws FileNotFoundException { + File dir0 = new File(testDir); + if (!dir0.exists()) { + throw new FileNotFoundException("Could not find test directory"); + } + File dir1 = new File(projectDir); + if (!dir1.exists()) { + if (!dir1.mkdir()) { + throw new FileNotFoundException("Could not create project directory"); + } + } + File dir2 = new File(xmlDir); + if (!dir2.exists()) { + if (!dir2.mkdir()) { + throw new FileNotFoundException("Could not create xml directory"); + } + } + File dir3 = new File(ghidraDir); + if (!dir3.exists()) { + if (!dir3.mkdir()) { + throw new FileNotFoundException("Could not create ghidra directory"); + } + } + } + + public void verifyRaw() throws IOException { + File rawDirectory = new File(rawDir); + if (!rawDirectory.exists()) { + throw new FileNotFoundException(rawDir); + } + if (!rawDirectory.isDirectory()) { + throw new FileNotFoundException("/raw is not a directory"); + } + String[] list = rawDirectory.list(); + boolean readlinePresent = false; + boolean historyPresent = false; + boolean bashPresent = false; + for (String element : list) { + File lib = new File(rawDirectory, element); + if (element.equals("libreadline.so.7.0")) { + String md5 = MD5Utilities.getMD5Hash(lib); + if (md5.equals(LIBREADLINE_MD5)) { + readlinePresent = true; + } + else { + throw new FileNotFoundException("libreadline.so.7.0 md5 does not match"); + } + } + else if (element.equals("libhistory.so.7.0")) { + String md5 = MD5Utilities.getMD5Hash(lib); + if (md5.equals(LIBHISTORY_MD5)) { + historyPresent = true; + } + else { + throw new FileNotFoundException("libhistory.so.7.0 md5 does not match"); + } + } + else if (element.equals("bash")) { + String md5 = MD5Utilities.getMD5Hash(lib); + if (md5.equals(BASH_MD5)) { + bashPresent = true; + } + else { + throw new FileNotFoundException("bash md5 does not match"); + } + } + } + if (!readlinePresent) { + throw new FileNotFoundException("Missing libreadline.so.7.0"); + } + if (!historyPresent) { + throw new FileNotFoundException("Missing libhistory.so.7.0"); + } + if (!bashPresent) { + throw new FileNotFoundException("Missing bash"); + } + } + + public void startServer() throws Exception { + if (isElasticSearch || isH2Database) { + return; // Don't try to start elasticsearch server + } + File touch = new File(serverTouchDir); + if (touch.exists()) { + return; + } + File dir = new File(serverDir); + if (dir.isDirectory()) { + FileUtilities.deleteDir(dir); + } + String[] params = new String[2]; + + params[0] = "start"; + params[1] = serverDir; + dir.mkdir(); // Create the data directory + + new BSimControlLaunchable().run(params); + + byte[] touchBytes = new byte[2]; + touchBytes[0] = 'a'; + touchBytes[1] = 'b'; + FileUtilities.writeBytes(touch, touchBytes); + } + + public void shutdownServer() throws Exception { + if (isElasticSearch || isH2Database) { + return; + } + File touch = new File(serverTouchDir); + if (!touch.exists()) { + return; + } + String[] params = new String[2]; + + params[0] = "stop"; + params[1] = serverDir; + new BSimControlLaunchable().run(params); + touch.delete(); // Remove the touch file + File dir = new File(serverDir); + if (dir.isDirectory()) { + FileUtilities.deleteDir(dir); // Clean up database files + } + } +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/client/BSimFunctionDatabaseTest.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/client/BSimFunctionDatabaseTest.java new file mode 100755 index 0000000000..8688318374 --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/client/BSimFunctionDatabaseTest.java @@ -0,0 +1,118 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.client; + +import static org.junit.Assert.*; + +import java.io.File; +import java.net.URL; +import java.util.ArrayList; +import java.util.List; + +import org.junit.Test; + +import generic.lsh.vector.LSHVector; +import generic.lsh.vector.LSHVectorFactory; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.facade.FunctionDatabaseTestDouble; +import ghidra.features.bsim.query.file.H2FileFunctionDatabase; +import ghidra.features.bsim.query.protocol.QueryNearest; +import ghidra.features.bsim.query.protocol.ResponseNearest; +import ghidra.test.AbstractGhidraHeadlessIntegrationTest; +import ghidra.util.Msg; + +public class BSimFunctionDatabaseTest extends AbstractGhidraHeadlessIntegrationTest { + + public BSimFunctionDatabaseTest() { + super(); + } + + /** + * Tests that we can recognize when a feature vector has already been used to query + * the db, so we don't query it again. + * + * To test, we create 5 of {@link FunctionDescription} instances, and populate each of them + * with one of two {@link LSHVector} instances. This should result in only 2 database queries, + * since we only have 2 unique vectors; the other 3 should be processed but NOT result in + * db queries. + * @throws Exception if an unexpected exception occurs + */ + @Test + public void testDupes() throws Exception { + + String dbPath = getTestDirectoryPath() + "/bsimTest" + BSimServerInfo.H2_FILE_EXTENSION; + URL dbUrl = new File(dbPath).toURI().toURL(); + + Msg.error(this, ">>>>>>>>>> Expected 2 Error Messages: \"...Database does not exist...\""); + + // First set up some objects we'll need to populate for the queries. + try (H2FileFunctionDatabase dbClient = new H2FileFunctionDatabase(dbUrl)) { + LSHVectorFactory vectorFactory = dbClient.getLSHVectorFactory(); + FunctionDatabaseTestDouble.loadWeightsFile(vectorFactory); + QueryNearest query = new QueryNearest(); + BSimSqlClause filter = null; + ResponseNearest response = new ResponseNearest(query); + DescriptionManager descMgr = new DescriptionManager(); + + //create a fake ExecutableRecord to avoid NPE + ExecutableRecord erec = new ExecutableRecord("name", "arch", new RowKeySQL(0)); + // Create a list of function descriptions; we'll loop over all of these, + // trying to query each one in turn. + List descs = new ArrayList<>(); + FunctionDescription desc1 = new FunctionDescription(erec, "d1", 0x10100); + FunctionDescription desc2 = new FunctionDescription(erec, "d2", 0x10200); + FunctionDescription desc3 = new FunctionDescription(erec, "d3", 0x10300); + FunctionDescription desc4 = new FunctionDescription(erec, "d4", 0x10400); + FunctionDescription desc5 = new FunctionDescription(erec, "d5", 0x10500); + + // Now create 2 different LSH vectors. Make them slightly different so + // we should treat them as distinct vectors. (Note that we already have tests + // that check vector equality in LSHVectorEqualityTest). + LSHVector vec1 = vectorFactory.buildVector(new int[] { 1, 2, 3 }); + LSHVector vec2 = vectorFactory.buildVector(new int[] { 1, 2, 4 }); + + // And now put those LSH vectors into SignatureRecords, which will be stored + // in the FunctionDescriptions. + SignatureRecord sigrec1 = new SignatureRecord(vec1); + SignatureRecord sigrec2 = new SignatureRecord(vec2); + + desc1.setSignatureRecord(sigrec1); + desc2.setSignatureRecord(sigrec1); + desc3.setSignatureRecord(sigrec2); + desc4.setSignatureRecord(sigrec1); + desc5.setSignatureRecord(sigrec2); + + descs.add(desc1); + descs.add(desc2); + descs.add(desc3); + descs.add(desc4); + descs.add(desc5); + + // Perform the query, which returns the number of unique vectors used to query + // the db. In this test it should have 2 entries. + int results = + dbClient.queryFunctions(query, filter, response, descMgr, descs.iterator()); + assertEquals(results, 2); + } + finally { + Msg.error(this, ">>>>>>>>>> End Expected Error Messages"); + File dbFile = new File(dbPath); + dbFile.delete(); + } + } + +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/FunctionDatabaseTestDouble.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/FunctionDatabaseTestDouble.java new file mode 100755 index 0000000000..760beba768 --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/FunctionDatabaseTestDouble.java @@ -0,0 +1,172 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import java.io.InputStream; +import java.net.MalformedURLException; +import java.net.URL; + +import generic.jar.ResourceFile; +import generic.lsh.vector.LSHVectorFactory; +import generic.lsh.vector.WeightedLSHCosineVectorFactory; +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.SQLFunctionDatabase; +import ghidra.features.bsim.query.description.DatabaseInformation; +import ghidra.features.bsim.query.protocol.BSimQuery; +import ghidra.features.bsim.query.protocol.QueryResponseRecord; +import ghidra.framework.Application; +import ghidra.framework.client.ClientUtil; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +public class FunctionDatabaseTestDouble implements SQLFunctionDatabase { + private static final String TEST_URL = "ghidra://localhost/db"; + + private Status status = Status.Ready; + private final String urlString; + private boolean canInitialize; + private String errorString; + private QueryResponseRecord record; + private static LSHVectorFactory vectorFactory = null; + + private BSimQuery lastQuery; + + public FunctionDatabaseTestDouble() { + this(TEST_URL); + } + + public FunctionDatabaseTestDouble(String urlString) { + this.urlString = urlString; + if (vectorFactory == null) { + vectorFactory = new WeightedLSHCosineVectorFactory(); + loadWeightsFile(vectorFactory); + } + } + + public static void loadWeightsFile(LSHVectorFactory factory) { + ResourceFile weightsFile = Application.findDataFileInAnyModule("lshweights_32.xml"); + try { + InputStream input = weightsFile.getInputStream(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(input, "Vector weights parser", + SpecXmlUtils.getXmlHandler(), false); + factory.readWeights(parser); + input.close(); + } + catch (Exception ex) { + // If weights aren't available, tests will fail reading settings + } + } + + @Override + public QueryResponseRecord query(BSimQuery query) { + this.status = Status.Busy; + this.lastQuery = query; + this.status = Status.Ready; + return record; + } + + BSimQuery getLastQuery() { + return lastQuery; + } + + public void setQueryResponse(QueryResponseRecord record) { + this.record = record; + } + + @Override + public Status getStatus() { + return status; + } + + void setStatus(Status status) { + this.status = status; + } + + @Override + public ConnectionType getConnectionType() { + return ConnectionType.Unencrypted_No_Authentication; + } + + @Override + public String getUserName() { + return ClientUtil.getUserName(); + } + + @Override + public void setUserName(String userName) { + // Currently not implemented + } + + @Override + public String getURLString() { + return urlString; + } + + @Override + public BSimServerInfo getServerInfo() { + try { + return new BSimServerInfo(new URL(urlString)); + } + catch (IllegalArgumentException | MalformedURLException e) { + throw new RuntimeException(e); + } + } + + @Override + public boolean initialize() { + return canInitialize; + } + + public void setCanInitialize(boolean canInitialize) { + this.canInitialize = canInitialize; + } + + @Override + public void close() { + // nothing to do + } + + @Override + public Error getLastError() { + return new Error(ErrorCategory.Unused, errorString); + } + + void setErrorString(String errorString) { + this.errorString = errorString; + } + + @Override + public DatabaseInformation getInfo() { + return new DatabaseInformation(); + } + + @Override + public int compareLayout() { + return 0; + } + + @Override + public LSHVectorFactory getLSHVectorFactory() { + return vectorFactory; + } + + @Override + public String formatBitAndSQL(String v1, String v2) { + return "(" + v1 + " & " + v2 + ")"; // copied from postgress + } + +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/SimilarFunctionQueryServiceTest.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/SimilarFunctionQueryServiceTest.java new file mode 100755 index 0000000000..a2a8b5c371 --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/SimilarFunctionQueryServiceTest.java @@ -0,0 +1,405 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import static org.junit.Assert.*; + +import java.util.*; + +import org.junit.*; + +import ghidra.features.bsim.gui.search.results.BSimMatchResult; +import ghidra.features.bsim.query.protocol.QueryResponseRecord; +import ghidra.features.bsim.query.protocol.ResponseNearest; +import ghidra.program.database.symbol.FunctionSymbol; +import ghidra.program.model.address.Address; +import ghidra.program.model.address.AddressFactory; +import ghidra.program.model.listing.*; +import ghidra.test.AbstractGhidraHeadlessIntegrationTest; +import ghidra.test.TestEnv; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; +import ghidra.util.task.TaskMonitorAdapter; + +public class SimilarFunctionQueryServiceTest extends AbstractGhidraHeadlessIntegrationTest { + + private TestEnv env; + private Program program; + SimilarFunctionQueryService queryService; + + public SimilarFunctionQueryServiceTest() { + super(); + } + + @Before + public void setUp() throws Exception { + env = new TestEnv(); + program = env.getProgram("notepad_w2k"); + } + + @After + public void tearDown() throws Exception { + if (queryService != null) { + queryService.dispose(); + } + env.release(program); + env.dispose(); + } + + @Test + public void testBuildQueryWithEmptyFunctions() throws Exception { + Set functions = new HashSet<>(); + try { + new SFQueryInfo(functions); + Assert.fail("Did not get expected exception passing an empty functions list"); + } + catch (IllegalArgumentException iae) { + // good! + } + } + + @Test + public void testQueryValidServerWithFunctions() throws Exception { + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + String serverURL = "ghidra://localhost/repo"; + + queryService.initializeDatabase(serverURL); + SFQueryResult result = + queryService.querySimilarFunctions(queryInfo, null, TaskMonitor.DUMMY); + assertNotNull(result); + + List similarFunctions = + BSimMatchResult.generate(result.getSimilarityResults(), program); + assertTrue(similarFunctions.isEmpty()); + } + + @Test + public void testQueryWithStagingOn() throws Exception { + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + queryService.setNumberOfStages(2); + String serverURL = "ghidra://localhost/repo"; + + queryService.initializeDatabase(serverURL); + SFQueryResult result = + queryService.querySimilarFunctions(queryInfo, null, TaskMonitor.DUMMY); + assertNotNull(result); + + List similarFunctions = + BSimMatchResult.generate(result.getSimilarityResults(), program); + assertTrue(similarFunctions.isEmpty()); + } + + @Test + public void testResultsListener() throws Exception { + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + + ResultsListener resultsListener = new ResultsListener(); + String serverURL = "ghidra://localhost/repo"; + + queryService.initializeDatabase(serverURL); + queryService.querySimilarFunctions(queryInfo, resultsListener, TaskMonitor.DUMMY); + + assertTrue("Did not receive a resultsAdded() callback", + resultsListener.resultsAddedCount >= 1); + assertTrue("Did not complete", resultsListener.operationComplete); + assertNotNull("Did not receive a final result", resultsListener.finalResult); + } + + @Test + public void testResultsListenerWithMultipleStagesGetsMultipleCallbacks() throws Exception { + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + + ResultsListener resultsListener = new ResultsListener(); + + int stageCount = 2; + queryService.setNumberOfStages(stageCount); + + String serverURL = "ghidra://localhost/repo"; + + queryService.initializeDatabase(serverURL); + queryService.querySimilarFunctions(queryInfo, resultsListener, TaskMonitor.DUMMY); + + assertEquals("Did not receive a resultsAdded() callback", stageCount, + resultsListener.resultsAddedCount); + assertTrue("Did not complete", resultsListener.operationComplete); + assertNotNull("Did not receive a final result", resultsListener.finalResult); + + resultsListener = new ResultsListener(); + + stageCount = 3; + + // Note: the staging code will not allow the number of stages to go past the number of functions + int actualStageCount = 2; + queryService.setNumberOfStages(stageCount); + + queryService.querySimilarFunctions(queryInfo, resultsListener, TaskMonitor.DUMMY); + + assertEquals("Did not receive a resultsAdded() callback", actualStageCount, + resultsListener.resultsAddedCount); + assertTrue("Did not complete", resultsListener.operationComplete); + assertNotNull("Did not receive a final result", resultsListener.finalResult); + } + + @Test + public void testMultipleQueriesToSameDatabaseGeneratesMultipleConnectedCallbacks() + throws Exception { + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + + ResultsListener resultsListener = new ResultsListener(); + String serverURL = "ghidra://localhost/repo"; + + queryService.initializeDatabase(serverURL); + + queryService.querySimilarFunctions(queryInfo, resultsListener, TaskMonitor.DUMMY); + assertTrue("Did not complete", resultsListener.operationComplete); + assertNotNull("Did not receive a final result", resultsListener.finalResult); + + resultsListener.reset(); + + queryService.querySimilarFunctions(queryInfo, resultsListener, TaskMonitor.DUMMY); + assertTrue("Did not complete", resultsListener.operationComplete); + assertNotNull("Did not receive a final result", resultsListener.finalResult); + } + + @Test + public void testQueryInvalidServer() { + // + // Make sure we get a connection error when attempting to connect when a server is not + // running. For this test, all we need to do is not supply a database test double. + // + + queryService = new SimilarFunctionQueryService(program); + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + String serverURL = "ghidra://localhost/repo"; + + try { + queryService.initializeDatabase(serverURL); + queryService.querySimilarFunctions(queryInfo, null, TaskMonitor.DUMMY); + Assert.fail("Did not receive an exception for failing to connect to the database: " + + serverURL); + } + catch (CancelledException ce) { + // shouldn't happen + } + catch (QueryDatabaseException qde) { + // good! + } + } + + @Test + public void testCreatingQueryInfo() { + try { + new SFQueryInfo(null); + Assert.fail("Did not get exception passing null for function list"); + } + catch (Exception e) { + // good! + } + + try { + new SFQueryInfo(new HashSet()); + Assert.fail("Did not get exception passing null for function list"); + } + catch (Exception e) { + // good! + } + } + + @Test + public void testInvalidUserMadeQuery() throws QueryDatabaseException { + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new BadQueryInfo(functions); + String serverURL = "ghidra://localhost/repo"; + + queryService.initializeDatabase(serverURL); + try { + queryService.querySimilarFunctions(queryInfo, null, TaskMonitor.DUMMY); + Assert.fail( + "Did not get an exception with a user-defined query that returns a null function list"); + } + catch (Exception e) { + // good! + } + } + + @Test + public void testCancelQuery() throws Exception { + // + // Rather than start multiple threads to test cancelling, we will just start the monitor + // in the cancelled state to simulate the user cancelling. + // + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + + ResultsListener resultsListener = new ResultsListener(); + String serverURL = "ghidra://localhost/repo"; + + TaskMonitorAdapter monitor = new TaskMonitorAdapter(); + monitor.setCancelEnabled(true); + monitor.cancel(); + + queryService.initializeDatabase(serverURL); + + try { + queryService.querySimilarFunctions(queryInfo, resultsListener, monitor); + Assert.fail("Did not get a cancelled exception as expected"); + } + catch (CancelledException ce) { + // good! + } + + assertEquals("Received a callback when the work should have been cancelled", 0, + resultsListener.resultsAddedCount); + assertTrue("Did not complete", resultsListener.operationComplete); + assertNull("Did not receive a final result", resultsListener.finalResult); + } + + @Test + public void testChangeDatabase() throws Exception { + // + // Test that passing a different database URL doesn't blow up (the database is cached, but + // should change when the user changes URLs) + // + createQueryService(); + + Set functions = createKnownFunctionsSet(); + SFQueryInfo queryInfo = new SFQueryInfo(functions); + String serverURL = "ghidra://localhost/repo"; + + queryService.initializeDatabase(serverURL); + SFQueryResult result = + queryService.querySimilarFunctions(queryInfo, null, TaskMonitor.DUMMY); + assertNotNull(result); + + List similarFunctions = + BSimMatchResult.generate(result.getSimilarityResults(), program); + assertTrue(similarFunctions.isEmpty()); + + serverURL = "ghidra://localhost/otherrepo"; + + try { + queryService.initializeDatabase(serverURL); + queryService.querySimilarFunctions(queryInfo, null, TaskMonitor.DUMMY); + } + catch (Exception e) { + // Expected--we can't connect to the newly supplied URL, as there is no server + // running on that port. However, the exception means that the code tried to change + // databases, which is what we wanted. + } + } + +//================================================================================================== +// Private Methods +//================================================================================================== + + private void createQueryService() { + assertNull(queryService); + String serverURL = "ghidra://localhost/repo"; + FunctionDatabaseTestDouble db = new FunctionDatabaseTestDouble(serverURL); + + // response + ResponseNearest response = new ResponseNearest(null); + db.setQueryResponse(response); + + queryService = new SimilarFunctionQueryService(program, db); + } + + private Set createKnownFunctionsSet() { + Set functions = new HashSet<>(); + FunctionManager functionManager = program.getFunctionManager(); + Function ghidra = functionManager.getFunctionAt(addr("010018a0")); + functions.add((FunctionSymbol) ghidra.getSymbol()); + Function sscanf = functionManager.getFunctionAt(addr("0100219c")); + functions.add((FunctionSymbol) sscanf.getSymbol()); + return functions; + } + + private Address addr(String addressString) { + AddressFactory factory = program.getAddressFactory(); + return factory.getAddress(addressString); + } + +//================================================================================================== +// Private Classes +//================================================================================================== + + private class ResultsListener implements SFResultsUpdateListener { + + private int resultsAddedCount; + private boolean operationComplete; + private SFQueryResult finalResult; + private List messages = new ArrayList<>(); + + @Override + public void resultAdded(QueryResponseRecord result) { + resultsAddedCount++; + assertNotNull(result); + } + + public void reset() { + resultsAddedCount = 0; + operationComplete = false; + finalResult = null; + messages.clear(); + } + +// @Override +// public void updateStatus(String message, MessageType type) { +// String msg = "Message(" + type + "): " + message; +// System.out.println(msg); +// messages.add(msg); +// } + + @Override + public void setFinalResult(SFQueryResult result) { + finalResult = result; + assertFalse("Operation previously completed", this.operationComplete); + this.operationComplete = true; + } + } + + private class BadQueryInfo extends SFQueryInfo { + + public BadQueryInfo(Set functions) { + super(functions); + } + + @Override + public Set getFunctions() { + return null; + } + } +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestBSimServerInfo.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestBSimServerInfo.java new file mode 100644 index 0000000000..d95cf4bd85 --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestBSimServerInfo.java @@ -0,0 +1,35 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.features.bsim.query.BSimServerInfo; +import ghidra.features.bsim.query.FunctionDatabase; + +public class TestBSimServerInfo extends BSimServerInfo { + + private FunctionDatabase database; + + public TestBSimServerInfo(FunctionDatabase database) { + super(DBType.postgres, "100.50.123.5", 123, "testDB"); + this.database = database; + } + + @Override + public FunctionDatabase getFunctionDatabase(boolean async) { + return database; + } + +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestNearestVectorResult.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestNearestVectorResult.java new file mode 100644 index 0000000000..2ae399c0b1 --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestNearestVectorResult.java @@ -0,0 +1,143 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import java.io.*; +import java.util.Date; +import java.util.List; + +import generic.lsh.vector.*; +import ghidra.features.bsim.query.description.*; +import ghidra.features.bsim.query.protocol.SimilarityVectorResult; +import ghidra.util.HashUtilities; +import ghidra.xml.XmlPullParser; + +public class TestNearestVectorResult extends SimilarityVectorResult { + + public TestNearestVectorResult(String queryFunctionName, String executableName, int hits, + double similarity) { + super(createFunctionDescription(queryFunctionName, executableName)); + VectorResult vectorResult = new VectorResult(1, hits, similarity, 0d, null); + addNotes(List.of(vectorResult)); + } + + protected static FunctionDescription createFunctionDescription(String queryFunctionName, + String executableName) { + String hash = getMd5(executableName); + ExecutableRecord executableRecord = + new ExecutableRecord(hash, executableName, "gcc", "x86", new Date(), null, null, null); + + FunctionDescription description = + new FunctionDescription(executableRecord, queryFunctionName, 0x10000); + description.setSignatureRecord(new SignatureRecord(new TestLSHVector())); + return description; + } + + private static String getMd5(String executableName) { + try { + InputStream is = new ByteArrayInputStream(executableName.getBytes()); + String hash = HashUtilities.getHash("MD5", is); + return hash; + } + catch (Exception e) { + return ""; + } + } + +// protected static double randomDouble() { +// return random.nextDouble(); +// } +// +// protected static double randomDoubleUnbounded() { +// return random.nextDouble() * random.nextInt(100); +// } + private static class TestLSHVector implements LSHVector { + + @Override + public int numEntries() { + return 7; + } + + @Override + public HashEntry getEntry(int i) { + // TODO Auto-generated method stub + return null; + } + + @Override + public HashEntry[] getEntries() { + // TODO Auto-generated method stub + return null; + } + + @Override + public double getLength() { + return 1d + Math.random(); + } + + @Override + public double compare(LSHVector op2, VectorCompare data) { + return 0; + } + + @Override + public void compareCounts(LSHVector op2, VectorCompare data) { + } + + @Override + public double compareDetail(LSHVector op2, StringBuilder buf) { + return 0; + } + + @Override + public void saveXml(Writer fwrite) throws IOException { + //stub + } + + @Override + public String saveSQL() { + return ""; + } + + @Override + public void saveBase64(StringBuilder buffer, char[] encoder) { + //stub + } + + @Override + public void restoreXml(XmlPullParser parser, WeightFactory weightFactory, + IDFLookup idfLookup) { + //stub + } + + @Override + public void restoreSQL(String sql, WeightFactory weightFactory, IDFLookup idfLookup) + throws IOException { + //stub + } + + @Override + public void restoreBase64(Reader input, char[] buffer, WeightFactory wfactory, + IDFLookup idflookup, int[] decode) throws IOException { + //stub + } + + @Override + public long calcUniqueHash() { + return 0; + } + } +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSFQueryService.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSFQueryService.java new file mode 100755 index 0000000000..c6c1ff2cc5 --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSFQueryService.java @@ -0,0 +1,48 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.features.bsim.query.protocol.ResponseNearestVector; +import ghidra.program.model.listing.Program; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +public class TestSFQueryService extends SimilarFunctionQueryService { + + private FunctionDatabase testDatabase; + + public TestSFQueryService(Program program, FunctionDatabase database) { + super(program, database); + this.testDatabase = database; + } + + @Override + public void initializeDatabase(String serverURLString) throws QueryDatabaseException { + return; + } + + @Override + public ResponseNearestVector overviewSimilarFunctions(SFOverviewInfo overviewInfo, + SFResultsUpdateListener listener, TaskMonitor monitor) + throws QueryDatabaseException, CancelledException { + + ResponseNearestVector response = (ResponseNearestVector) testDatabase.query(null); + listener.setFinalResult(response); + return response; + } + +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSFQueryServiceFactory.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSFQueryServiceFactory.java new file mode 100755 index 0000000000..0e17ae4c0b --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSFQueryServiceFactory.java @@ -0,0 +1,34 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import ghidra.features.bsim.query.FunctionDatabase; +import ghidra.program.model.listing.Program; + +public class TestSFQueryServiceFactory extends SFQueryServiceFactory { + + private FunctionDatabase database; + + public TestSFQueryServiceFactory(FunctionDatabase functionDatabase) { + database = functionDatabase; + } + + @Override + public SimilarFunctionQueryService createSFQueryService(Program program) { + return new TestSFQueryService(program, database); + } + +} diff --git a/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSimilarityResult.java b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSimilarityResult.java new file mode 100755 index 0000000000..30ae1dddc5 --- /dev/null +++ b/Ghidra/Features/BSim/src/test/java/ghidra/features/bsim/query/facade/TestSimilarityResult.java @@ -0,0 +1,67 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.features.bsim.query.facade; + +import java.io.ByteArrayInputStream; +import java.io.InputStream; +import java.util.Date; + +import ghidra.features.bsim.query.description.ExecutableRecord; +import ghidra.features.bsim.query.description.FunctionDescription; +import ghidra.features.bsim.query.protocol.SimilarityResult; +import ghidra.util.HashUtilities; + +public class TestSimilarityResult extends SimilarityResult { +// protected static Random random = new Random(); + + public TestSimilarityResult(String queryFunctionName, String executableName, + String matchFunction, long address, double significance, double confidence) { + super(createFunctionDescription(queryFunctionName, executableName, address)); + addNote(createFunctionDescription(matchFunction, executableName, address), significance, + confidence); + } + + protected static FunctionDescription createFunctionDescription(String queryFunctionName, + String executableName, long address) { + String hash = getMd5(executableName); + ExecutableRecord executableRecord = + new ExecutableRecord(hash, executableName, "gcc", "x86", new Date(), null, null, null); + + FunctionDescription description = + new FunctionDescription(executableRecord, queryFunctionName, address); + return description; + } + + private static String getMd5(String executableName) { + try { + InputStream is = new ByteArrayInputStream(executableName.getBytes()); + String hash = HashUtilities.getHash("MD5", is); + return hash; + } + catch (Exception e) { + return ""; + } + } + +// protected static double randomDouble() { +// return random.nextDouble(); +// } +// +// protected static double randomDoubleUnbounded() { +// return random.nextDouble() * random.nextInt(100); +// } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/Module.manifest b/Ghidra/Features/BSimFeatureVisualizer/Module.manifest new file mode 100755 index 0000000000..64cba6ebf1 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/Module.manifest @@ -0,0 +1 @@ +##MODULE IP: Nuvola Icons - LGPL 2.1 diff --git a/Ghidra/Features/BSimFeatureVisualizer/build.gradle b/Ghidra/Features/BSimFeatureVisualizer/build.gradle new file mode 100755 index 0000000000..5884a312a7 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/build.gradle @@ -0,0 +1,28 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +apply from: "$rootProject.projectDir/gradle/distributableGhidraModule.gradle" +apply from: "$rootProject.projectDir/gradle/javaProject.gradle" +apply from: "$rootProject.projectDir/gradle/javaTestProject.gradle" +apply from: "$rootProject.projectDir/gradle/helpProject.gradle" + +apply plugin: 'eclipse' +eclipse.project.name = 'Features BSimFeatureVisualizer' + +dependencies { + api project(':Decompiler') + api project(':BSim') +} + diff --git a/Ghidra/Features/BSimFeatureVisualizer/certification.manifest b/Ghidra/Features/BSimFeatureVisualizer/certification.manifest new file mode 100755 index 0000000000..6665759d74 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/certification.manifest @@ -0,0 +1,5 @@ +##VERSION: 2.0 +Module.manifest||GHIDRA||||END| +data/bsim.theme.properties||GHIDRA||||END| +src/main/help/help/TOC_Source.xml||GHIDRA||||END| +src/main/help/help/topics/BSimFeatureVisualizerPlugin/BSimFeatureVisualizerPlugin.htm||GHIDRA||||END| diff --git a/Ghidra/Features/BSimFeatureVisualizer/data/bsim.theme.properties b/Ghidra/Features/BSimFeatureVisualizer/data/bsim.theme.properties new file mode 100644 index 0000000000..5522a9f796 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/data/bsim.theme.properties @@ -0,0 +1,29 @@ + +[Defaults] + +color.bsim.graph.vertex.default = color.palette.lightgray +color.bsim.graph.edge.default = color.palette.black +color.bsim.graph.vertex.selection = color.palette.gold +color.bsim.graph.edge.selection = color.palette.gold + +color.bsim.graph.dataflow.vertex.base = color.palette.black +color.bsim.graph.dataflow.vertex.base.2 = color.palette.darkgray +color.bsim.graph.dataflow.vertex.pcode.op = color.palette.red +color.bsim.graph.dataflow.vertex.varnode.collapsed = color.palette.lightskyblue +color.bsim.graph.dataflow.vertex.pcode.op.collapsed = color.palette.lightskyblue + +color.bsim.graph.controlflow.vertex.base = color.palette.aliceblue +color.bsim.graph.controlflow.vertex.parent = color.palette.lightgreen +color.bsim.graph.controlflow.vertex.grandparent = color.palette.darkgreen +color.bsim.graph.controlflow.vertex.child = color.palette.purple +color.bsim.graph.controlflow.vertex.sibling = color.palette.blue +color.bsim.graph.controlflow.vertex.neighbor = color.palette.cornsilk + +color.bsim.graph.edge.dataflow.in = color.palette.lemonchiffon +color.bsim.graph.edge.dataflow.out = color.palette.saddlebrown +color.bsim.graph.edge.dataflow.in.collapsed = color.palette.lightgray +color.bsim.graph.edge.dataflow.out.collapsed = color.palette.darkgray +color.bsim.graph.edge.controlflow.true = color.palette.green +color.bsim.graph.edge.controlflow.false = color.palette.red + +[Dark Defaults] diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/help/help/TOC_Source.xml b/Ghidra/Features/BSimFeatureVisualizer/src/main/help/help/TOC_Source.xml new file mode 100644 index 0000000000..b277c9a980 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/help/help/TOC_Source.xml @@ -0,0 +1,57 @@ + + + + + + + + diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/help/help/topics/BSimFeatureVisualizerPlugin/BSimFeatureVisualizerPlugin.htm b/Ghidra/Features/BSimFeatureVisualizer/src/main/help/help/topics/BSimFeatureVisualizerPlugin/BSimFeatureVisualizerPlugin.htm new file mode 100644 index 0000000000..6f130e1627 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/help/help/topics/BSimFeatureVisualizerPlugin/BSimFeatureVisualizerPlugin.htm @@ -0,0 +1,137 @@ + + + + + + + + BSim Feature Visualizer Plugin + + + + +

BSim Feature Visualizer Plugin

+ +

The BSim Feature Visualizer Plugin is used to visualize the + BSim Features of the function at the current address. + +

The BSim Feature Table

+ +

The features are displayed in the BSim Feature Table. In order to + explain the columns of the table, we first briefly discuss BSim feature types.

+ +

DATA_FLOW features

+

These features describe data flow into a single "base" varnode. A neighborhood of the data + flow graph consisting of the base varnode and all vertices reachable by traversing edges + against the direction of flow (through at most 3 pcode ops) is hashed to produce these + features. + +

CONTROL_FLOW features

+

These features describe small neighborhoods of the control flow graph. Each neighborhood + is defined by a "base" block.

+ +

COMBINED features

+

These features are generated from a neighborhood in the control flow graph and data flow + information into the first "root" pcode op in the block, which must be either a CALL, CALLIND, + CALLOTHER, STORE, CBRANCH, BRANCHIND, or RETURN op.

+ +

DUAL_FLOW features

+

These features are formed from two adjacent "root" ops (as defined above) in a single basic + block.

+ +

COPY_SIG features

+

These features are formed from the "standalone copies" within a basic block. An example of + a standalone copy is an assignment of a constant to a global variable with no further dataflow + within the function.

+ +

Table Columns

+

We now list the columns of the BSim feature table. Note that some columns only apply to + some BSim feature types. If a column does not apply to a given feature it will be blank the in + corresponding row.

+
    +
  • Sequence Number: + (See here + for details on sequence numbers).For DATA_FLOW and COMBINED features, + this column contains + the sequence number of the pcode op associated with the feature. For DUAL_FLOW features it + contains the sequence number of the main pcode op (which is the second op in address order). + For CONTROL_FLOW features it contains an artificial sequence number corresponding to the + beginning of the associated basic block.
    • +
  • Address: This column is hidden by default. It contains the address corresponding to + the sequence number defined above.
    • +
  • Base Varnode: For DATA_FLOW features, this column shows the corresponding base + varnode. For other feature types it is blank.
    • +
  • Basic Block Start: For CONTROL_FLOW and COMBINED features this is the start address + of the basic block corresponding to the feature. For DATA_FLOW and DUAL_FLOW features it is + the start address of the basic block containing the relevant pcode op(s).
    • +
  • Block Index: This is the block index of the basic block defined above.
    • +
  • Feature: This is the 32-bit hash corresponding to the feature.
    • +
  • Feature Type: The feature type of the feature.
    • +
  • Pcode Op Name: For DATA_FLOW features this is the name of the pcode op defining the + base varnode. For COMBINED features it is the root pcode op, and for DUAL_FLOW features it is + the main pcode op.
    • +
  • Previous Op Info: For DUAL_FLOW features, this is the Mnemonic and Sequence number + of the previous pcode op.
    • +
+ +

Visualizing BSim Features

+

To visualize a BSim feature, right-click on a row in the BSim Feature Table and select + "Highlight and Graph". This action creates a graph showing the regions of the control flow and + data flow graphs which correspond to the feature. Most (but not all) of the data that goes + into the hash is depicted in the graph. The action also highlights some tokens in the + decompiler associated with the feature. Note that if the "Highlight by Row" option is + selected then the decompiler highlight is applied automatically whenever the selected + row changes. + +

The Graph

+

Hovering over a vertex or edge in the graph will display a popup listing its attributes. + Vertices corresponding to basic blocks are displayed as rectangle whereas vertices + corresponding to varnodes and pcode ops are generally drawn as circles (function inputs and + constants are drawn as triangles). Certain COPY, MULTIEQUAL, and INDIRECT ops are colored + differently to indicate that they are collapsed during feature generation. Note that the graph + can be used to navigate and make selections. + +

Decompiler Highlights

+

Certain tokens in the decompiler corresponding to the feature are highlighted in the + decompiler. In general, the highlight show less information than the graph. For instance, + for DATA_FLOW features, only the base varnode, its defining op, and the line containing them + are highlighted. Note that sometimes there will not be a token to highlight, which can result + from features of temporary varnodes that are not associated with a C language token. Also note + that the decompiler performs additional transformations when producing C code which can + complicate token highlighting. + +

Removing Decompiler Highlights

+

Click the in the upper right corner of the table to clear any + decompiler highlights added by this plugin. Other decompiler highlights are unaffected.

+ +

Options

+ +

This plugin has several options which can be set in the Tool Options menu.

+ +

Database Configuration File

+ +

This file must be a BSim database configuration template. An "index tuning" parameter is + read from this file and passed to the decompiler when generating BSim features. See + here + for details.

+ +

Reuse Graph

+ +

If checked, the same graph window will be used to display all BSim feature graphs. Only + one feature graph is displayed at a time (the previous feature graph will be cleared). If + unchecked, each feature graph will be drawn in a separate window. Note that in either case only + one feature at a time will be highlighted in the decompiler.

+ +

Decompiler Timeout

+ +

This parameter controls the maximum amount of time (in seconds) the decompiler will spend + trying to decompile a function before quitting.

+ +

Highlight by Row

+ +

If selected, the decompiler highlights will be applied whenever the selected row in the + BSim Feature Table changes.

+ + + diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureGraphDisplayOptions.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureGraphDisplayOptions.java new file mode 100644 index 0000000000..8824a88766 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureGraphDisplayOptions.java @@ -0,0 +1,79 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import docking.Tool; +import ghidra.service.graph.*; +import ghidra.util.HelpLocation; + +/** + * This class is the {@link GraphDisplayOptions} for BSim Feature Graphs. + */ +public class BSimFeatureGraphDisplayOptions extends GraphDisplayOptions { + + public BSimFeatureGraphDisplayOptions(GraphType graphType, Tool tool) { + super(graphType, tool, + new HelpLocation("BSimFeatureVisualizerPlugin", "Visualizing_BSim_Features")); + } + + @Override + protected void initializeDefaults() { + setDefaultVertexColor("color.bsim.graph.vertex.default"); + setDefaultEdgeColor("color.bsim.graph.edge.default"); + setVertexSelectionColor("color.bsim.graph.vertex.selection"); + setEdgeSelectionColor("color.bsim.graph.edge.selection"); + + setDefaultVertexShape(VertexShape.ELLIPSE); + setDefaultLayoutAlgorithmName("Hierarchical MinCross Top Down"); + setUsesIcons(false); + setLabelPosition(GraphLabelPosition.EAST); + + configureVertexType(BSimFeatureGraphType.PCODE_OP_VERTEX, VertexShape.ELLIPSE, + "color.bsim.graph.dataflow.vertex.pcode.op"); + configureVertexType(BSimFeatureGraphType.BASE_VARNODE_VERTEX, VertexShape.ELLIPSE, + "color.bsim.graph.dataflow.vertex.base"); + configureVertexType(BSimFeatureGraphType.SECONDARY_BASE_VARNODE_VERTEX, VertexShape.ELLIPSE, + "color.bsim.graph.dataflow.vertex.base.2"); + configureVertexType(BSimFeatureGraphType.COLLAPSED_OP, VertexShape.ELLIPSE, + "color.bsim.graph.dataflow.vertex.pcode.op.collapsed"); + configureVertexType(BSimFeatureGraphType.COLLAPSED_VARNODE, VertexShape.ELLIPSE, + "color.bsim.graph.dataflow.vertex.varnode.collapsed"); + + configureVertexType(BSimFeatureGraphType.BASE_BLOCK_VERTEX, VertexShape.RECTANGLE, + "color.bsim.graph.controlflow.vertex.base"); + configureVertexType(BSimFeatureGraphType.PARENT_BLOCK_VERTEX, VertexShape.RECTANGLE, + "color.bsim.graph.controlflow.vertex.parent"); + configureVertexType(BSimFeatureGraphType.GRANDPARENT_BLOCK_VERTEX, VertexShape.RECTANGLE, + "color.bsim.graph.controlflow.vertex.grandparent"); + configureVertexType(BSimFeatureGraphType.SIBLING_BLOCK_VERTEX, VertexShape.RECTANGLE, + "color.bsim.graph.controlflow.vertex.sibling"); + configureVertexType(BSimFeatureGraphType.CHILD_BLOCK_VERTEX, VertexShape.RECTANGLE, + "color.bsim.graph.controlflow.vertex.child"); + configureVertexType(BSimFeatureGraphType.BSIM_NEIGHBOR_VERTEX, VertexShape.RECTANGLE, + "color.bsim.graph.controlflow.vertex.neighbor"); + + configureEdgeType(BSimFeatureGraphType.TRUE_EDGE, "color.bsim.graph.edge.controlflow.true"); + configureEdgeType(BSimFeatureGraphType.FALSE_EDGE, + "color.bsim.graph.edge.controlflow.false"); + configureEdgeType(BSimFeatureGraphType.COLLAPSED_IN, + "color.bsim.graph.edge.dataflow.in.collapsed"); + configureEdgeType(BSimFeatureGraphType.COLLAPSED_OUT, + "color.bsim.graph.edge.dataflow.out.collapsed"); + configureEdgeType(BSimFeatureGraphType.DATAFLOW_IN, "color.bsim.graph.edge.dataflow.in"); + configureEdgeType(BSimFeatureGraphType.DATAFLOW_OUT, "color.bsim.graph.edge.dataflow.out"); + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureGraphType.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureGraphType.java new file mode 100644 index 0000000000..69bd7ff287 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureGraphType.java @@ -0,0 +1,126 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.util.ArrayList; +import java.util.List; + +import ghidra.service.graph.GraphType; + +/** + * This class is the {@link GraphType} for BSim Feature Graphs. + */ +public class BSimFeatureGraphType extends GraphType { + private static List vertexTypes = new ArrayList<>(); + private static List edgeTypes = new ArrayList<>(); + + //dataflow vertex types + public static final String DEFAULT_VERTEX = "Default"; + public static final String CONSTANT_VERTEX = "Constant"; + public static final String VARNODE_ADDRESS = "Address Varnode"; + public static final String BASE_VARNODE_VERTEX = "Base Varnode"; + public static final String SECONDARY_BASE_VARNODE_VERTEX = "Secondary Base Varnode"; + public static final String FUNCTION_INPUT = "Function Input"; + public static final String CONSTANT_FUNCTION_INPUT = "Constant Function Input"; + public static final String PCODE_OP_VERTEX = "Pcode Op"; + public static final String VOID_BASE = "void"; + public static final String COLLAPSED_VARNODE = "Collapsed Varnode"; + public static final String COLLAPSED_OP = "Collapsed Op"; + + //dataflow vertex attributes + public static final String OP_ADDRESS = "Address"; + public static final String PCODE_OUTPUT = "Pcode Output"; + public static final String SIZE = "Size"; + + //dataflow edge types + public static final String DATAFLOW_IN = "Input"; + public static final String DATAFLOW_OUT = "Output"; + public static final String COLLAPSED_IN = "Collapsed Input"; + public static final String COLLAPSED_OUT = "Collapsed Output"; + + //control flow vertex types + public static final String BASE_BLOCK_VERTEX = "Base Block"; + public static final String PARENT_BLOCK_VERTEX = "Parent Block"; + public static final String GRANDPARENT_BLOCK_VERTEX = "Grandparent Block"; + public static final String CHILD_BLOCK_VERTEX = "Child Block"; + public static final String SIBLING_BLOCK_VERTEX = "Sibling Block"; + public static final String NULL_BLOCK_VERTEX = "Null Block"; + + //for blocks that can't be categorized cleanly within a bsim neighborhood using + //ancestor/descendant relations + public static final String BSIM_NEIGHBOR_VERTEX = "BSim Neighbor Block"; + + //control flow vertex attributes + public static final String BLOCK_START = "Block Start"; + public static final String BLOCK_STOP = "Block Stop"; + public static final String CALL_STRING = "Call String"; + public static final String EMPTY_CALL_STRING = "(empty)"; + + //control flow edge types + public static final String TRUE_EDGE = "True"; + public static final String FALSE_EDGE = "False"; + public static final String CONTROL_FLOW_DEFAULT_EDGE = "Default"; + + //copy signature attributes + public static final String COPY_SIGNATURE = "Copy Signature"; + + public static int DATAFLOW_WINDOW_SIZE = 3; + public static final String DATAFLOW_PREFIX = "df"; + public static final String CONTROL_FLOW_PREFIX = "cf"; + public static final String COPY_PREFIX = "copy"; + + public static final String OPTIONS_NAME = "BSim Feature Graph"; + + static { + vertexTypes.add(DEFAULT_VERTEX); + vertexTypes.add(CONSTANT_VERTEX); + vertexTypes.add(VARNODE_ADDRESS); + vertexTypes.add(BASE_VARNODE_VERTEX); + vertexTypes.add(SECONDARY_BASE_VARNODE_VERTEX); + vertexTypes.add(FUNCTION_INPUT); + vertexTypes.add(CONSTANT_FUNCTION_INPUT); + vertexTypes.add(PCODE_OP_VERTEX); + vertexTypes.add(VOID_BASE); + vertexTypes.add(COLLAPSED_VARNODE); + vertexTypes.add(COLLAPSED_OP); + vertexTypes.add(BASE_BLOCK_VERTEX); + vertexTypes.add(PARENT_BLOCK_VERTEX); + vertexTypes.add(GRANDPARENT_BLOCK_VERTEX); + vertexTypes.add(CHILD_BLOCK_VERTEX); + vertexTypes.add(SIBLING_BLOCK_VERTEX); + vertexTypes.add(NULL_BLOCK_VERTEX); + vertexTypes.add(BSIM_NEIGHBOR_VERTEX); + + edgeTypes.add(DATAFLOW_IN); + edgeTypes.add(DATAFLOW_OUT); + edgeTypes.add(COLLAPSED_IN); + edgeTypes.add(COLLAPSED_OUT); + edgeTypes.add(TRUE_EDGE); + edgeTypes.add(FALSE_EDGE); + edgeTypes.add(CONTROL_FLOW_DEFAULT_EDGE); + + } + + public BSimFeatureGraphType() { + super("BSim Feature Graph", "BSim Feature Graph", vertexTypes, edgeTypes); + } + + @Override + public String getOptionsName() { + return OPTIONS_NAME; + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureType.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureType.java new file mode 100755 index 0000000000..25ee59e84e --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureType.java @@ -0,0 +1,35 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +public enum BSimFeatureType { + // Signature describes data-flow to a single varnode + DATA_FLOW, + + // Signature describes control-flow for a basic-block + CONTROL_FLOW, + + // Signature describes control-flow for a basic-block and + // data-flow into the first (root) PcodeOp in the block + COMBINED, + + // Signature describes data-flow into two root PcodeOps + // that are adjacent in a basic-block + DUAL_FLOW, + + // Signature describes stand-alone COPY ops within a single block + COPY_SIG +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureVisualizerPlugin.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureVisualizerPlugin.java new file mode 100755 index 0000000000..98e7882128 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BSimFeatureVisualizerPlugin.java @@ -0,0 +1,283 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.io.IOException; + +import org.xml.sax.SAXException; + +import docking.action.builder.ActionBuilder; +import generic.jar.ResourceFile; +import ghidra.app.decompiler.DecompilerHighlightService; +import ghidra.app.events.ProgramLocationPluginEvent; +import ghidra.app.events.ProgramSelectionPluginEvent; +import ghidra.app.plugin.ProgramPlugin; +import ghidra.app.services.GoToService; +import ghidra.app.services.GraphDisplayBroker; +import ghidra.features.bsim.query.BsimPluginPackage; +import ghidra.features.bsim.query.client.Configuration; +import ghidra.framework.Application; +import ghidra.framework.model.DomainObjectChangedEvent; +import ghidra.framework.model.DomainObjectListener; +import ghidra.framework.options.OptionsChangeListener; +import ghidra.framework.options.ToolOptions; +import ghidra.framework.plugintool.PluginInfo; +import ghidra.framework.plugintool.PluginTool; +import ghidra.framework.plugintool.util.PluginStatus; +import ghidra.program.model.listing.*; +import ghidra.program.util.ProgramLocation; +import ghidra.util.HelpLocation; +import ghidra.util.Msg; + +//@formatter:off +@PluginInfo( + status = PluginStatus.RELEASED, + packageName = BsimPluginPackage.NAME, + category = "BSim", + shortDescription = "BSim Feature Visualizer", + description = "Displays BSim features as graphs and highlighted regions in the decompiler.", + servicesRequired = { GoToService.class, GraphDisplayBroker.class, + DecompilerHighlightService.class}, + eventsProduced = { ProgramLocationPluginEvent.class, ProgramSelectionPluginEvent.class } +) +//@formatter:on + +/** + * A plugin for visualizing BSim features as graphs and highlights in the decompiler. + */ +public class BSimFeatureVisualizerPlugin extends ProgramPlugin + implements DomainObjectListener, OptionsChangeListener { + + private BsfvTableProvider provider; + private Function currentFunction; + + //options + public static final String BSIM_FEATURE_VISUALIZER_OPTION_NAME = "BsimFeatureVisualizer"; + public static final String DB_CONFIG_FILE = "Database Configuration File"; + public static final String REUSE_GRAPH = "Reuse Graph"; + public static final String DECOMPILER_TIMEOUT = "Decompiler Timeout"; + public static final String HIGHLIGHT_BY_ROW = "Highlight by Row"; + public static final String BSIM_FEATURE_VISUALIZER_ACTION = "Show BSim Feature Visualizer"; + private String dbConfigFile = "medium_nosize.xml"; + private boolean reuseGraph = true; + private boolean highlightByRow = true; + private int decompilerTimeout = 10; + private DecompilerHighlightService highlightService; + + /** + * Creates a BSimFeatureVisualizerPlugin for the given {@link PluginTool} + * @param tool plugin tool + */ + public BSimFeatureVisualizerPlugin(PluginTool tool) { + super(tool); + } + + @Override + public void init() { + initOptions(tool.getOptions(BSIM_FEATURE_VISUALIZER_OPTION_NAME)); + highlightService = getTool().getService(DecompilerHighlightService.class); + + new ActionBuilder(BSIM_FEATURE_VISUALIZER_ACTION, getName()) + .menuPath("BSim", "BSim Feature Visualizer") + .helpLocation(new HelpLocation(getName(), getName())) + .onAction(c -> { + if (currentLocation != null) { + FunctionManager functionManager = currentProgram.getFunctionManager(); + currentFunction = + functionManager.getFunctionContaining(currentLocation.getAddress()); + } + provider = new BsfvTableProvider(this); + }) + .buildAndInstall(tool); + } + + @Override + public void optionsChanged(ToolOptions options, String optionName, Object oldValue, + Object newValue) { + switch (optionName) { + case DB_CONFIG_FILE: + dbConfigFile = (String) newValue; + if (provider != null) { + provider.reload(); + } + break; + case REUSE_GRAPH: + reuseGraph = (Boolean) newValue; + break; + case DECOMPILER_TIMEOUT: + decompilerTimeout = (Integer) newValue; + if (provider != null) { + provider.reload(); + } + break; + case HIGHLIGHT_BY_ROW: + highlightByRow = (Boolean) newValue; + break; + default: + Msg.error(this, "Unrecognized option: " + optionName); + break; + } + } + + @Override + protected void locationChanged(ProgramLocation location) { + if (provider == null) { + return; + } + + if (location == null) { + return; + } + if (currentFunction != null && currentFunction.getBody().contains(location.getAddress())) { + return; + } + FunctionManager functionManager = currentProgram.getFunctionManager(); + currentFunction = functionManager.getFunctionContaining(location.getAddress()); + if (currentFunction == null) { + return; + } + provider.reload(); + } + + @Override + protected void programActivated(Program program) { + if (provider == null) { + return; + } + program.addListener(this); + if (currentLocation != null) { + FunctionManager functionManager = currentProgram.getFunctionManager(); + currentFunction = functionManager.getFunctionContaining(currentLocation.getAddress()); + } + provider.programOpened(program); + } + + @Override + protected void programDeactivated(Program program) { + currentFunction = null; + program.removeListener(this); + if (provider != null) { + provider.programDeactivated(); + } + } + + /** + * Returns the function whose features are in displayed in the table + * @return current function + */ + Function getFunction() { + return currentFunction; + } + + @Override + public void domainObjectChanged(DomainObjectChangedEvent ev) { + //Following {@link DecompilerProvider}'s lead, reload on any change to the program + //note that BSimFeatureTableProvider.reload() checks for visibility + if (provider != null) { + provider.reload(); + } + } + + @Override + public void dispose() { + if (currentProgram != null) { + currentProgram.removeListener(this); + } + if (provider != null) { + provider.dispose(); + } + super.dispose(); + } + + /** + * Parses the signature settings from the database configuration file specified in the options + * for this plugin. + * @return signature settings + */ + int getSignatureSettings() { + ResourceFile dbConfigurationFile = Application.findDataFileInAnyModule(dbConfigFile); + if (dbConfigurationFile == null) { + Msg.showError(this, null, "File not found", "Couldn't find file " + dbConfigFile); + return 0; + } + Configuration dbConfig = new Configuration(); + + try { + //load template appends ".xml" so strip it off here + dbConfig.loadTemplate(dbConfigurationFile.getParentFile(), + dbConfigFile.substring(0, dbConfigFile.length() - 4)); + } + catch (SAXException | IOException e) { + Msg.showError(dbConfig, null, "Problem with configuration file " + dbConfigFile, + e.getMessage()); + } + return dbConfig.info.settings; + } + + /** + * Returns a boolean determining whether the plugin should reuse the graph when + * drawing feature graphs. + * @return reuseGraph + */ + public boolean getReuseGraph() { + return reuseGraph; + } + + /** + * Returns the decompiler timeout setting. + * @return decompiler timeout + */ + public int getDecompilerTimeout() { + return decompilerTimeout; + } + + /** + * Returns a boolean indicating whether the plugin should automatically apply decompiler + * highlights when the selected row changes. + * @return highlight by row + */ + public boolean getHighlightByRow() { + return highlightByRow; + } + + /** + * Returns the {@link DecompilerHighlightService} for this plugin. + * @return decompiler highlight service + */ + DecompilerHighlightService getDecompilerHighlightService() { + return highlightService; + } + + private void initOptions(ToolOptions options) { + options.registerOption(DB_CONFIG_FILE, dbConfigFile, + new HelpLocation(this.getName(), "Config_File"), + "Database configuration file to read signature settings from."); + dbConfigFile = options.getString(DB_CONFIG_FILE, dbConfigFile); + options.registerOption(REUSE_GRAPH, reuseGraph, + new HelpLocation(this.getName(), "Reuse_Graph"), + "Clear and re-use the graph window or create new graph window when graphing features."); + reuseGraph = options.getBoolean(REUSE_GRAPH, reuseGraph); + options.registerOption(DECOMPILER_TIMEOUT, decompilerTimeout, + new HelpLocation(this.getName(), "Decompiler_Timeout"), "Decompiler Timeout (seconds)"); + decompilerTimeout = options.getInt(DECOMPILER_TIMEOUT, decompilerTimeout); + options.registerOption(HIGHLIGHT_BY_ROW, highlightByRow, + new HelpLocation(this.getName(), "Highlight_By_Row"), + "Highlight feature in decompiler whenever selected row changes"); + highlightByRow = options.getBoolean(HIGHLIGHT_BY_ROW, highlightByRow); + options.addOptionsChangeListener(this); + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvFeatureColumnObject.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvFeatureColumnObject.java new file mode 100644 index 0000000000..18386d773e --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvFeatureColumnObject.java @@ -0,0 +1,45 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +/** + * This class is used to display the actual BSim feature in the BSim feature table. Formally, + * a BSim feature is a 32 bit hash. We wrap such an integer with an instance of this class in + * order to force a particular unsigned comparison and hexadecimal display in the feature table. + */ +public class BsfvFeatureColumnObject implements Comparable { + + private int bsimFeature; + + /** + * Creates a BSimFeatureColumnType corresponding to the BSim feature with the given hash + * @param hash feature value + */ + public BsfvFeatureColumnObject(int hash) { + bsimFeature = hash; + } + + @Override + public String toString() { + return Integer.toHexString(bsimFeature); + } + + @Override + public int compareTo(BsfvFeatureColumnObject o) { + return Integer.compareUnsigned(bsimFeature, o.bsimFeature); + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvGraphDisplayListener.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvGraphDisplayListener.java new file mode 100644 index 0000000000..5868ed6227 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvGraphDisplayListener.java @@ -0,0 +1,108 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.util.HashSet; +import java.util.Set; + +import ghidra.app.plugin.core.graph.AddressBasedGraphDisplayListener; +import ghidra.framework.plugintool.PluginTool; +import ghidra.program.model.address.*; +import ghidra.program.model.listing.Program; +import ghidra.service.graph.*; + +/** + * This class is the {@link AddressBasedGraphDisplayListener} for translating between vertices of + * the BSim feature graphs and Ghidra addresses. + */ +public class BsfvGraphDisplayListener extends AddressBasedGraphDisplayListener { + + /** + * Creates a BSimFeatureGraphDisplayListener used to translate between vertices of a BSim + * feature graph and Ghidra addresses + * @param tool tool containing the bsim feature visualizer plugin + * @param program source of BSim features + * @param display graph display for showing the feature graphs + */ + public BsfvGraphDisplayListener(PluginTool tool, Program program, GraphDisplay display) { + super(tool, program, display); + } + + @Override + public GraphDisplayListener cloneWith(GraphDisplay gDisplay) { + return new BsfvGraphDisplayListener(tool, program, gDisplay); + } + + @Override + protected Set getVertices(AddressSetView selection) { + Set vertices = new HashSet<>(); + AddressFactory addrFactory = program.getAddressFactory(); + for (AttributedVertex v : graphDisplay.getGraph().vertexSet()) { + if (v.hasAttribute(BSimFeatureGraphType.OP_ADDRESS)) { + Address opAddr = + addrFactory.getAddress(v.getAttribute(BSimFeatureGraphType.OP_ADDRESS)); + if (selection.contains(opAddr)) { + vertices.add(v); + } + } + if (v.hasAttribute(BSimFeatureGraphType.BLOCK_START) && + v.hasAttribute(BSimFeatureGraphType.BLOCK_STOP)) { + Address start = + addrFactory.getAddress(v.getAttribute(BSimFeatureGraphType.BLOCK_START)); + Address stop = + addrFactory.getAddress(v.getAttribute(BSimFeatureGraphType.BLOCK_STOP)); + if (selection.intersects(start, stop)) { + vertices.add(v); + } + } + } + return vertices; + } + + @Override + protected AddressSet getAddresses(Set vertexIds) { + AddressSet addresses = new AddressSet(); + AddressFactory addrFactory = program.getAddressFactory(); + for (AttributedVertex v : vertexIds) { + if (v.hasAttribute(BSimFeatureGraphType.OP_ADDRESS)) { + addresses.add( + addrFactory.getAddress(v.getAttribute(BSimFeatureGraphType.OP_ADDRESS))); + } + if (v.hasAttribute(BSimFeatureGraphType.BLOCK_START) && + v.hasAttribute(BSimFeatureGraphType.BLOCK_STOP)) { + Address start = + addrFactory.getAddress(v.getAttribute(BSimFeatureGraphType.BLOCK_START)); + Address stop = + addrFactory.getAddress(v.getAttribute(BSimFeatureGraphType.BLOCK_STOP)); + addresses.add(start, stop); + } + } + return addresses; + } + + @Override + protected Address getAddress(AttributedVertex vertex) { + AddressFactory addrFactory = program.getAddressFactory(); + if (vertex.hasAttribute(BSimFeatureGraphType.OP_ADDRESS)) { + return addrFactory.getAddress(vertex.getAttribute(BSimFeatureGraphType.OP_ADDRESS)); + } + if (vertex.hasAttribute(BSimFeatureGraphType.BLOCK_START)) { + return addrFactory.getAddress(vertex.getAttribute(BSimFeatureGraphType.BLOCK_START)); + } + return null; + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvRowObject.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvRowObject.java new file mode 100755 index 0000000000..b0401c479e --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvRowObject.java @@ -0,0 +1,145 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import ghidra.program.model.address.Address; +import ghidra.program.model.pcode.*; + +/** + * The class represents a row in the table of BSim features + */ +public class BsfvRowObject { + + private BsfvFeatureColumnObject feature; + private SequenceNumber seq; + private PcodeOpAST definingPcodeOp; + private PcodeOpAST previousPcodeOp; + private BSimFeatureType type; + private Varnode baseVarnode; + private Address basicBlockStart; + private Integer blockIndex; + + /** + * Creates a row object for a table of BSim features. Note that not all columns are appropriate + * for all feature types. For CONTROL_FLOW features, {@code seq} should be an artificial + * {@link SequenceNumber} corresponding to the start of the appropriate basic block. + * @param feature hash value (required) + * @param seq sequence number of feature (required) + * @param baseVarnode base varnode of DATA_FLOW signature (null otherwise) + * @param definingPcodeOp defining op of feature (null for CONTROL_FLOW features) + * @param previousPcodeOp previous op (only non-null for DUAL_FLOW features) + * @param type BSimFeatureType of feature (required) + * @param basicBlockStart start of basic block (null for DATA_FLOW signatures) + * @param blockIndex index of basic block (null for DATA_FLOW signatures) + */ + public BsfvRowObject(int feature, SequenceNumber seq, Varnode baseVarnode, + PcodeOpAST definingPcodeOp, PcodeOpAST previousPcodeOp, BSimFeatureType type, + Address basicBlockStart, Integer blockIndex) { + this.feature = new BsfvFeatureColumnObject(feature); + this.seq = seq; + this.type = type; + this.baseVarnode = baseVarnode; + this.basicBlockStart = basicBlockStart; + this.definingPcodeOp = definingPcodeOp; + this.previousPcodeOp = previousPcodeOp; + this.blockIndex = blockIndex; + } + + /** + * Returns the {@SequenceNumber} corresponding to the feature. + * @return sequence number + */ + public SequenceNumber getSeq() { + return seq; + } + + /** + * Returns the PcodeOpAST corresponding to the feature. + * @return pcodeop ast + */ + public PcodeOpAST getPcodeOpAST() { + return definingPcodeOp; + } + + /** + * Returns the previous PcodeOpAST correspond to the features. Only non-null for DUAL_FLOW + * features. + * @return previous pcodeop ast + */ + public PcodeOpAST getPreviousPcodeOpAST() { + return previousPcodeOp; + } + + /** + * Returns the mnemonic of the defining pcode op. Returns null if there is no defining pcode op. + * @return mnemonic of defining op + */ + public String getOpMnemonic() { + if (definingPcodeOp != null) { + return definingPcodeOp.getMnemonic(); + } + return null; + } + + /** + * Returns the {@link BSimFeatureType} of the feature. + * @return bsim feature type + */ + public BSimFeatureType getBSimFeatureType() { + return type; + } + + /** + * Returns the base + * @return base varnode + */ + public Varnode getBaseVarnode() { + return baseVarnode; + } + + /** + * Returns the start of the basic block corresponding to the feature + * @return basic block start + */ + public Address getBasicBlockStart() { + return basicBlockStart; + } + + /** + * Returns the {@link Address} corresponding to the feature + * @return address of feature + */ + public Address getAddress() { + return seq.getTarget(); + } + + /** + * Returns the {@BSimFeatureColumnType} wrapping the 32-bit hash + * @return wrapped hash + */ + public BsfvFeatureColumnObject getFeature() { + return feature; + } + + /** + * Returns the index of the basic block corresponding to the features + * @return basic block index + */ + public Integer getBlockIndex() { + return blockIndex; + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTableModel.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTableModel.java new file mode 100755 index 0000000000..720df5dd8f --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTableModel.java @@ -0,0 +1,471 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.util.*; + +import docking.widgets.table.AbstractDynamicTableColumn; +import docking.widgets.table.TableColumnDescriptor; +import ghidra.app.decompiler.*; +import ghidra.app.decompiler.signature.*; +import ghidra.docking.settings.Settings; +import ghidra.framework.plugintool.ServiceProvider; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.program.model.pcode.*; +import ghidra.util.Msg; +import ghidra.util.datastruct.Accumulator; +import ghidra.util.exception.CancelledException; +import ghidra.util.table.AddressBasedTableModel; +import ghidra.util.task.TaskMonitor; + +class BsfvTableModel extends AddressBasedTableModel { + + private BSimFeatureVisualizerPlugin plugin; + private HighFunction hfunction; + private Map seqToPcode; + private Map blockIndexToCallString; + private ArrayList signatures; + private Set featuredOps; + + public static boolean DEBUG = false; + + public BsfvTableModel(BSimFeatureVisualizerPlugin plugin, Program program) { + super("BSim Feature Visualizer", plugin.getTool(), program, null); + this.plugin = plugin; + } + + @Override + protected TableColumnDescriptor createTableColumnDescriptor() { + TableColumnDescriptor descriptor = new TableColumnDescriptor<>(); + descriptor.addHiddenColumn(new AddressTableColumn()); + descriptor.addVisibleColumn(new OpSequenceNumberTableColumn(), 1, true); + descriptor.addVisibleColumn(new BSimFeatureTableColumn(), 2, true); + descriptor.addVisibleColumn(new BSimFeatureTypeTableColumn()); + descriptor.addVisibleColumn(new PcodeOpNameTableColumn()); + descriptor.addVisibleColumn(new BaseVarnodeTableColumn()); + descriptor.addVisibleColumn(new BasicBlockAddressTableColumn()); + descriptor.addVisibleColumn(new PreviousOpInfoTableColumn()); + descriptor.addHiddenColumn(new BasicBlockIndexColumn()); + return descriptor; + } + + @Override + protected void doLoad(Accumulator accumulator, TaskMonitor monitor) + throws CancelledException { + + if (plugin.getCurrentProgram() == null) { + return; + } + + Function function = plugin.getFunction(); + if (function == null) { + return; // not inside a function + } + + if (!decompile(function, monitor)) { + return; + } + + featuredOps = new HashSet<>(); + //process the debug signatures: + //tie them to the pcodeopast/basic blocks from the high function + //create row of table for each feature + for (int i = 0; i < signatures.size(); ++i) { + int hash = signatures.get(i).hash; + SequenceNumber seq = null; + BSimFeatureType type = null; + Varnode vn = null; + PcodeOpAST pcode = null; + PcodeOpAST previousPcode = null; + Address basicBlockStart = null; + Integer blockIndex = null; + if (signatures.get(i) instanceof VarnodeSignature varSig) { + seq = varSig.seqNum; + type = BSimFeatureType.DATA_FLOW; + vn = varSig.vn; + pcode = seqToPcode.get(varSig.seqNum); + if (pcode != null && pcode.getParent() != null) { + basicBlockStart = + pcode.getParent().getIterator().next().getSeqnum().getTarget(); + featuredOps.add(pcode); + } + if (pcode == null) { + Msg.info(this, "null pcode for feature " + hash + ", op = " + varSig.opcode + + ", seq = " + seq); + } + else { + if (!pcode.getMnemonic().equals(varSig.opcode)) { + Msg.info(this, "op mis-match at " + seq + ", varSig = " + varSig.opcode + + ", pcode = " + pcode.getMnemonic()); + } + } + } + else if (signatures.get(i) instanceof CopySignature copySig) { + blockIndex = copySig.index; + type = BSimFeatureType.COPY_SIG; + basicBlockStart = hfunction.getBasicBlocks().get(blockIndex).getStart(); + seq = new SequenceNumber(basicBlockStart, 0); + } + else { + BlockSignature blockSig = (BlockSignature) signatures.get(i); + basicBlockStart = blockSig.blockSeq; + blockIndex = blockSig.index; + if (blockSig.opSeq == null) { + // If opSeq is null, then previousOpSeq is null as well + seq = new SequenceNumber(blockSig.blockSeq, 0); + // This is a pure control-flow feature + type = BSimFeatureType.CONTROL_FLOW; + basicBlockStart = blockSig.blockSeq; + } + else if (blockSig.previousOpSeq == null) { // If we only have the primary op + seq = blockSig.opSeq; + // This is the first root op, mixed with control-flow info + type = BSimFeatureType.COMBINED; + basicBlockStart = blockSig.blockSeq; + pcode = seqToPcode.get(blockSig.opSeq); + } + else { + seq = blockSig.opSeq; + // This is two consecutive root ops, mixed together + type = BSimFeatureType.DUAL_FLOW; + basicBlockStart = blockSig.blockSeq; + pcode = seqToPcode.get(blockSig.opSeq); + previousPcode = seqToPcode.get(blockSig.previousOpSeq); + } + } + accumulator.add(new BsfvRowObject(hash, seq, vn, pcode, previousPcode, type, + basicBlockStart, blockIndex)); + if (DEBUG) { + Msg.debug(this, i + ": " + seq + " " + Integer.toUnsignedString(hash, 16)); + } + } + } + + @Override + public Address getAddress(int row) { + return getRowObject(row).getAddress(); + } + + /** + * Returns the {@link PcodeOpAST} object at the specified row. + * @param row row index + * @return pcodeop + */ + public PcodeOpAST getOpAt(int row) { + return getRowObject(row).getPcodeOpAST(); + } + + /** + * Returns the previous {@link PcodeOpAST} object at the specified row. + * @param row row index + * @return previous pcodeop + */ + public PcodeOpAST getPreviousOpAt(int row) { + return getRowObject(row).getPreviousPcodeOpAST(); + } + + /** + * Returns the {@link BSimFeatureType} at the specified row + * @param row row index + * @return bsim feature type + */ + public BSimFeatureType getFeatureTypeAt(int row) { + return getRowObject(row).getBSimFeatureType(); + } + + /** + * Returns the basic block index at the specified row + * @param row row index + * @return basic block index + */ + public Integer getBlockIndexAt(int row) { + return getRowObject(row).getBlockIndex(); + } + + /** + * Returns the start address of the basic block at the specified row + * @param row row index + * @return start of block + */ + public Address getBasicBlockStart(int row) { + return getRowObject(row).getBasicBlockStart(); + } + + /** + * Returns the {@link HighFunction} of the function whose features currently populate the table. + * @return high function + */ + public HighFunction getHighFunction() { + return hfunction; + } + + /** + * Returns an unmodifiable view of the set of pcode ops whose outputs are the base varnodes + * of DATA_FLOW features. + * @return pcode ops corresponding to DATA_FLOW features + */ + public Set getFeaturedOps() { + return Collections.unmodifiableSet(featuredOps); + } + + /** + * Returns the "call string" of a basic block, which represents the number and ordering + * of CALL and CALLIND pcode ops with the block. + * @param index block index + * @return call string + */ + public String getCallString(int index) { + return blockIndexToCallString.get(index); + } + + /** + * Sets the current program and reloads the table data. + * @param p program + */ + public void reload(Program p) { + setProgram(p); + reload(); + } + + @Override + public void setProgram(Program program) { + if (this.program != program) { + this.program = program; + clearData(); + } + } + + private boolean decompile(Function function, TaskMonitor monitor) { + + DecompInterface decompiler = null; + try { + decompiler = getConfiguredDecompiler(); + if (!decompiler.openProgram(plugin.getCurrentProgram())) { + Msg.info(this, "Unable to initalize the Decompiler interface"); + Msg.info(this, decompiler.getLastMessage()); + return false; + } + int decompilerTimeout = plugin.getDecompilerTimeout(); + //first decompile the function to get the HighFunction + DecompileResults decompRes = + decompiler.decompileFunction(function, decompilerTimeout, monitor); + hfunction = decompRes.getHighFunction(); + if (hfunction == null) { + Msg.info(this, "null HighFunction for " + function.getName()); + return false; + } + //populate the map from block indices to call strings and + //create a map from SequenceNumbers to PcodeOps + //needed since the DebugSignature objects only contain the sequence number + seqToPcode = new HashMap<>(); + blockIndexToCallString = new HashMap<>(); + for (PcodeBlockBasic block : hfunction.getBasicBlocks()) { + StringBuilder sb = new StringBuilder(); + Iterator pcodeOpIter = block.getIterator(); + while (pcodeOpIter.hasNext()) { + PcodeOpAST op = (PcodeOpAST) pcodeOpIter.next(); + seqToPcode.put(op.getSeqnum(), op); + if ((op.getOpcode() == PcodeOp.CALL) || (op.getOpcode() == PcodeOp.CALLIND)) { + if (sb.length() > 0) { + sb.append(","); + } + sb.append(op.getMnemonic()); + } + } + if (sb.length() == 0) { + sb.append(BSimFeatureGraphType.EMPTY_CALL_STRING); + } + blockIndexToCallString.put(block.getIndex(), sb.toString()); + } + + //next use the decompiler to get the debug BSim feature information + signatures = decompiler.debugSignatures(function, decompilerTimeout, null); + if (signatures == null) { + Msg.info(this, "Null sigres for function " + function.getName()); + return false; + } + } + finally { + if (decompiler != null) { + decompiler.closeProgram(); + decompiler.dispose(); + } + } + return true; + } + + private DecompInterface getConfiguredDecompiler() { + DecompInterface decompiler = new DecompInterface(); + decompiler.setOptions(new DecompileOptions()); + decompiler.toggleSyntaxTree(true); + decompiler.setSimplificationStyle("normalize"); + decompiler.setSignatureSettings(plugin.getSignatureSettings()); + return decompiler; + } + + //============================================================================================== + // Inner Classes (Table Columns) + //============================================================================================== + + private class PreviousOpInfoTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Previous Op Info"; + } + + @Override + public String getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider sProvider) throws IllegalArgumentException { + PcodeOp prev = rowObject.getPreviousPcodeOpAST(); + if (prev == null) { + return null; + } + return prev.getMnemonic() + ": " + prev.getSeqnum().toString(); + } + + } + + private class PcodeOpNameTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Pcode Op Name"; + } + + @Override + public String getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider services) throws IllegalArgumentException { + return rowObject.getOpMnemonic(); + } + } + + private class BaseVarnodeTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Base Varnode"; + } + + @Override + public String getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider services) throws IllegalArgumentException { + if (rowObject.getBaseVarnode() != null) { + return rowObject.getBaseVarnode().toString(program.getLanguage()); + } + return null; + } + } + + private class BSimFeatureTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Feature"; + } + + @Override + public BsfvFeatureColumnObject getValue(BsfvRowObject rowObject, Settings settings, + Object data, ServiceProvider services) throws IllegalArgumentException { + return rowObject.getFeature(); + } + } + + private class OpSequenceNumberTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Op Sequence Number"; + } + + @Override + public SequenceNumber getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider sProvider) throws IllegalArgumentException { + return rowObject.getSeq(); + } + + } + + private class BasicBlockAddressTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Basic Block Start"; + } + + @Override + public Address getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider sProvider) throws IllegalArgumentException { + return rowObject.getBasicBlockStart(); + } + + } + + private class AddressTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Address"; + } + + @Override + public Address getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider services) throws IllegalArgumentException { + return rowObject.getAddress(); + } + } + + private class BSimFeatureTypeTableColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Feature Type"; + } + + @Override + public String getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider sProvider) throws IllegalArgumentException { + return rowObject.getBSimFeatureType().toString(); + } + + } + + private class BasicBlockIndexColumn + extends AbstractDynamicTableColumn { + + @Override + public String getColumnName() { + return "Block Index"; + } + + @Override + public Integer getValue(BsfvRowObject rowObject, Settings settings, Object data, + ServiceProvider sProvider) throws IllegalArgumentException { + return rowObject.getBlockIndex(); + } + + } +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTableProvider.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTableProvider.java new file mode 100755 index 0000000000..72f731ebd7 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTableProvider.java @@ -0,0 +1,215 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.awt.*; + +import javax.swing.*; + +import ghidra.app.decompiler.DecompilerHighlightService; +import ghidra.app.decompiler.DecompilerHighlighter; +import ghidra.framework.plugintool.ComponentProviderAdapter; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.program.util.ProgramLocation; +import ghidra.util.HelpLocation; +import ghidra.util.table.*; + +/** + * This class is the {@link ComponentProviderAdapter} for the BSim feature table. + */ +public class BsfvTableProvider extends ComponentProviderAdapter { + + private BSimFeatureVisualizerPlugin plugin; + private JComponent component; + private GhidraTable bsimFeatureTable; + private GhidraThreadedTablePanel bsimFeatureTablePanel; + private BsfvTableModel model; + private GhidraTableFilterPanel tableFilterPanel; + private JPanel panel; + private Component table; + private ClearDecompilerHighlightsAction clearingAction; + + /** + * Creates a provider for the BSim feature table. + * @param plugin owning plugin + */ + public BsfvTableProvider(BSimFeatureVisualizerPlugin plugin) { + super(plugin.getTool(), "BSimFeatureVisualizer", plugin.getName()); + this.plugin = plugin; + component = build(); + setTransient(); + tool.addComponentProvider(this, true); + setHelpLocation(new HelpLocation(plugin.getName(), plugin.getName())); + HighlightAndGraphAction highlightAndGraphAction = new HighlightAndGraphAction(this, plugin); + tool.addLocalAction(this, highlightAndGraphAction); + clearingAction = new ClearDecompilerHighlightsAction(plugin); + tool.addLocalAction(this, clearingAction); + } + + @Override + public JComponent getComponent() { + return component; + } + + /** + * Removes any decompiler highlights generated by this plugin, reloads the table, and sets the + * title. + */ + public void reload() { + if (isVisible()) { + clearingAction.clearHighlights(); + model.reload(plugin.getCurrentProgram()); + setTitle(getTitleText()); + } + } + + /** + * Reloads the table and sets the title + * @param program new program + */ + public void programOpened(Program program) { + if (isVisible()) { + model.reload(program); + setTitle(getTitleText()); + } + } + + /** + * Clears any decompiler highlights associated with this plugin, empties the table, and resets + * the table title. + */ + public void programDeactivated() { + clearingAction.clearHighlights(); + model.reload(null); + setTitle(getTitleText()); + + } + + /** + * Removes {@code this} from the tool. + */ + void dispose() { + tool.removeComponentProvider(this); + } + + @Override + public void componentShown() { + model.reload(plugin.getCurrentProgram()); + } + + @Override + public void componentHidden() { + clearingAction.clearHighlights(); + } + + /** + * Returns the underlying {@link GhidraTable} of the BSim feature table + * @return table + */ + GhidraTable getTable() { + return bsimFeatureTable; + } + + /** + * Returns the BSim feature table model + * @return table model + */ + BsfvTableModel getModel() { + return model; + } + + private String getTitleText() { + String title = "BSim Feature Visualizer"; + if (plugin.getCurrentProgram() == null) { + return title; + } + ProgramLocation loc = plugin.getProgramLocation(); + if (loc == null) { + return title; //can happen when switching between tabbed programs in the listing + } + Function func = + plugin.getCurrentProgram().getFunctionManager().getFunctionContaining(loc.getAddress()); + if (func == null) { + return title; + } + return title + ": " + func.getName(); + } + + private JComponent build() { + panel = new JPanel(new BorderLayout()); + table = buildTablePanel(); + panel.add(table, BorderLayout.CENTER); + return panel; + } + + private Component buildTablePanel() { + + model = new BsfvTableModel(plugin, plugin.getCurrentProgram()); + bsimFeatureTablePanel = new GhidraThreadedTablePanel<>(model, 1000); + bsimFeatureTable = bsimFeatureTablePanel.getTable(); + bsimFeatureTable.setName("BSim Features"); + + bsimFeatureTable.installNavigation(tool); + bsimFeatureTable.setNavigateOnSelectionEnabled(true); + bsimFeatureTable.setAutoResizeMode(JTable.AUTO_RESIZE_SUBSEQUENT_COLUMNS); + bsimFeatureTable.setPreferredScrollableViewportSize(new Dimension(900, 300)); + bsimFeatureTable.setRowSelectionAllowed(true); + bsimFeatureTable.setSelectionMode(ListSelectionModel.SINGLE_SELECTION); + + bsimFeatureTable.getSelectionModel().addListSelectionListener(e -> { + if (!plugin.getHighlightByRow()) { + return; + } + DecompilerHighlightService service = plugin.getDecompilerHighlightService(); + if (service == null) { + return; + } + int selectedRow = bsimFeatureTable.getSelectedRow(); + if (selectedRow == -1) { + return; + } + BsfvRowObject row = model.getRowObject(selectedRow); + BsfvTokenHighlightMatcher tokenMatcher = + new BsfvTokenHighlightMatcher(row, model.getHighFunction(), plugin); + DecompilerHighlighter highlighter = service.createHighlighter( + HighlightAndGraphAction.BSIM_FEATURE_HIGHLIGHTER_NAME, tokenMatcher); + highlighter.applyHighlights(); + }); + + model.addTableModelListener(e -> { + int rowCount = model.getRowCount(); + int unfilteredCount = model.getUnfilteredRowCount(); + + StringBuilder buffy = new StringBuilder(); + + buffy.append(rowCount).append(" items"); + if (rowCount != unfilteredCount) { + buffy.append(" (of ").append(unfilteredCount).append(" )"); + } + + setSubTitle(buffy.toString()); + }); + + tableFilterPanel = new GhidraTableFilterPanel<>(bsimFeatureTable, model); + + JPanel container = new JPanel(new BorderLayout()); + container.add(bsimFeatureTablePanel, BorderLayout.CENTER); + container.add(tableFilterPanel, BorderLayout.SOUTH); + return container; + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTokenHighlightMatcher.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTokenHighlightMatcher.java new file mode 100644 index 0000000000..a1d7dc7693 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/BsfvTokenHighlightMatcher.java @@ -0,0 +1,208 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.awt.Color; +import java.util.*; + +import generic.theme.GThemeDefaults.Colors.Palette; +import generic.theme.Gui; +import ghidra.app.decompiler.*; +import ghidra.app.decompiler.component.DecompilerUtils; +import ghidra.framework.options.Options; +import ghidra.program.model.address.AddressSet; +import ghidra.program.model.pcode.*; + +/** + * This class is used to highlight tokens in the decompiler corresponding to BSim features + */ +class BsfvTokenHighlightMatcher implements CTokenHighlightMatcher { + + private static final Color DEFAULT_HIGHLIGHT = Palette.ORANGE; + private static final Color LINE_HIGHLIGHT_COLOR = Palette.getColor("lightskyblue"); + private static final Color SECONDARY_LINE_HIGHLIGHT_COLOR = Palette.getColor("steelblue"); + + private PcodeOpAST pcodeOp; + private PcodeOpAST previousPcodeOp; + private PcodeBlockBasic block; + private Set lineHighlights; + private Set secondaryLineHighlights; + private Set blockHighlights; + private Options graphOptions; + + public BsfvTokenHighlightMatcher(BsfvRowObject row, HighFunction highFunction, + BSimFeatureVisualizerPlugin plugin) { + graphOptions = + plugin.getTool().getOptions("Graph").getOptions(BSimFeatureGraphType.OPTIONS_NAME); + switch (row.getBSimFeatureType()) { + case DATA_FLOW: + this.pcodeOp = row.getPcodeOpAST(); + break; + case COPY_SIG: + //just highlight block + //could improve by finding the standalone copies and only highlighting them + case CONTROL_FLOW: + this.block = highFunction.getBasicBlocks().get(row.getBlockIndex()); + break; + case COMBINED: + this.block = highFunction.getBasicBlocks().get(row.getBlockIndex()); + this.pcodeOp = row.getPcodeOpAST(); + break; + case DUAL_FLOW: + this.block = highFunction.getBasicBlocks().get(row.getBlockIndex()); + this.pcodeOp = row.getPcodeOpAST(); + this.previousPcodeOp = row.getPreviousPcodeOpAST(); + break; + default: + throw new IllegalArgumentException( + "Unsupported feature type: " + row.getBSimFeatureType().toString()); + } + } + + /** + * Creates a highlighter for DATA_FLOW features + * @param pcodeOp defining pcode op + */ + public BsfvTokenHighlightMatcher(PcodeOpAST pcodeOp) { + this.pcodeOp = pcodeOp; + } + + /** + * Creates a highlighter for CONTROL_FLOW features + * @param block base block + */ + public BsfvTokenHighlightMatcher(PcodeBlockBasic block) { + this.block = block; + } + + /** + * Creates a highlighter for COMBINED features + * @param pcodeOp root op + * @param block root block + */ + public BsfvTokenHighlightMatcher(PcodeOpAST pcodeOp, PcodeBlockBasic block) { + this.pcodeOp = pcodeOp; + this.block = block; + } + + /** + * Creates a highlighter for DUAL_FLOW features + * @param pcodeOp pcode op + * @param previousPcodeOp previous pcode op + */ + public BsfvTokenHighlightMatcher(PcodeOpAST pcodeOp, PcodeOpAST previousPcodeOp) { + this.pcodeOp = pcodeOp; + this.previousPcodeOp = previousPcodeOp; + } + + @Override + public void start(ClangNode root) { + lineHighlights = new HashSet<>(); + secondaryLineHighlights = new HashSet<>(); + blockHighlights = new HashSet<>(); + + if (pcodeOp != null) { + List opTokens = + DecompilerUtils.getTokens(root, pcodeOp.getSeqnum().getTarget()); + for (ClangToken token : opTokens) { + lineHighlights.add(token.getLineParent().getLineNumber()); + } + } + + if (previousPcodeOp != null) { + List secondaryOpTokens = + DecompilerUtils.getTokens(root, previousPcodeOp.getSeqnum().getTarget()); + for (ClangToken token : secondaryOpTokens) { + secondaryLineHighlights.add(token.getLineParent().getLineNumber()); + } + } + + if (block != null) { + AddressSet blockRange = new AddressSet(block.getStart(), block.getStop()); + List tokensInBlock = DecompilerUtils.getTokens(root, blockRange); + for (ClangToken token : tokensInBlock) { + ClangLine line = token.getLineParent(); + if (line != null) { + blockHighlights.add(line.getLineNumber()); + } + } + } + } + + @Override + public Color getTokenHighlight(ClangToken token) { + Options options = graphOptions.getOptions("Vertex Colors"); + String opKey = BSimFeatureGraphType.PCODE_OP_VERTEX; + Color color = options.getColor(opKey, DEFAULT_HIGHLIGHT); + if (token instanceof ClangFuncNameToken) { + PcodeOp op = token.getPcodeOp(); + if (op == null) { + return null; + } + if (pcodeOp != null && op.getSeqnum().equals(pcodeOp.getSeqnum())) { + return color; + } + if (previousPcodeOp != null && op.getSeqnum().equals(previousPcodeOp.getSeqnum())) { + return Gui.darker(color); + } + } + if (token instanceof ClangOpToken) { + PcodeOp op = token.getPcodeOp(); + if (op == null) { + return null; + } + if (pcodeOp != null && op.getSeqnum().equals(pcodeOp.getSeqnum())) { + return color; + } + if (previousPcodeOp != null && op.getSeqnum().equals(previousPcodeOp.getSeqnum())) { + return Gui.darker(color); + } + } + if (token instanceof ClangVariableToken) { + ClangVariableToken varToken = (ClangVariableToken) token; + PcodeOp op = varToken.getPcodeOp(); + Varnode vnode = varToken.getVarnode(); + if (op == null) { + return null; + } + if (pcodeOp != null && op.getSeqnum().equals(pcodeOp.getSeqnum())) { + if (vnode != null && vnode.equals(op.getOutput())) { + return options.getColor(BSimFeatureGraphType.BASE_VARNODE_VERTEX, + DEFAULT_HIGHLIGHT); + } + } + if (previousPcodeOp != null && op.getSeqnum().equals(previousPcodeOp.getSeqnum())) { + if (vnode != null && vnode.equals(op.getOutput())) { + return options.getColor(BSimFeatureGraphType.SECONDARY_BASE_VARNODE_VERTEX, + DEFAULT_HIGHLIGHT); + } + } + } + if (token.getLineParent() != null) { + if (lineHighlights.contains(token.getLineParent().getLineNumber())) { + return LINE_HIGHLIGHT_COLOR; + } + if (secondaryLineHighlights.contains(token.getLineParent().getLineNumber())) { + return SECONDARY_LINE_HIGHLIGHT_COLOR; + } + if (blockHighlights.contains(token.getLineParent().getLineNumber())) { + return options.getColor(BSimFeatureGraphType.BASE_BLOCK_VERTEX, DEFAULT_HIGHLIGHT); + } + } + return null; + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/ClearDecompilerHighlightsAction.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/ClearDecompilerHighlightsAction.java new file mode 100644 index 0000000000..29d2ba994c --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/ClearDecompilerHighlightsAction.java @@ -0,0 +1,71 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.awt.Color; + +import docking.ActionContext; +import docking.action.DockingAction; +import docking.action.ToolBarData; +import ghidra.app.decompiler.*; +import ghidra.util.HelpLocation; +import ghidra.util.Msg; +import resources.Icons; + +/** + * This action is used to remove any decompiler highlights added by the + * {@BSimFeatureVisualizerPlugin}. + */ +public class ClearDecompilerHighlightsAction extends DockingAction { + BSimFeatureVisualizerPlugin plugin; + + public ClearDecompilerHighlightsAction(BSimFeatureVisualizerPlugin plugin) { + super("Clear Decompiler Highlights", plugin.getName()); + this.plugin = plugin; + this.setToolBarData(new ToolBarData(Icons.DELETE_ICON)); + setDescription("Remove decompiler highlights"); + setHelpLocation(new HelpLocation(plugin.getName(), "Removing_Decompiler_Highlights")); + } + + @Override + public void actionPerformed(ActionContext context) { + clearHighlights(); + } + + /** + * Clears any decompiler highlights associated with {@link BSimFeatureVisualizerPlugin}. + */ + void clearHighlights() { + DecompilerHighlightService service = + plugin.getTool().getService(DecompilerHighlightService.class); + if (service == null) { + Msg.showError(this, null, "DecompilerHighlightService not found", + "DecompilerHighlightService not found."); + return; + } + DecompilerHighlighter highlighter = service.createHighlighter( + HighlightAndGraphAction.BSIM_FEATURE_HIGHLIGHTER_NAME, new ClearingHighlightMatcher()); + highlighter.applyHighlights(); + } + + private class ClearingHighlightMatcher implements CTokenHighlightMatcher { + @Override + public Color getTokenHighlight(ClangToken token) { + return null; + } + } + +} diff --git a/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/HighlightAndGraphAction.java b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/HighlightAndGraphAction.java new file mode 100644 index 0000000000..ced0db6867 --- /dev/null +++ b/Ghidra/Features/BSimFeatureVisualizer/src/main/java/ghidra/bsfv/HighlightAndGraphAction.java @@ -0,0 +1,488 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.bsfv; + +import java.util.*; + +import docking.ActionContext; +import docking.action.DockingAction; +import docking.action.MenuData; +import ghidra.app.decompiler.DecompilerHighlightService; +import ghidra.app.decompiler.DecompilerHighlighter; +import ghidra.app.services.GraphDisplayBroker; +import ghidra.program.model.address.Address; +import ghidra.program.model.pcode.*; +import ghidra.service.graph.*; +import ghidra.util.HelpLocation; +import ghidra.util.Msg; +import ghidra.util.exception.CancelledException; +import ghidra.util.exception.GraphException; +import ghidra.util.table.GhidraTable; +import ghidra.util.task.TaskMonitor; + +/** + * This action is used to draw BSim feature graphs and to determine which tokens in the decompiler + * should be highlighted for a given feature. + */ +public class HighlightAndGraphAction extends DockingAction { + + public static final String BSIM_FEATURE_HIGHLIGHTER_NAME = "BSimFeatureHighlighter"; + public static final String NAME = "Highlight and Graph"; + + private BsfvTableProvider provider; + private BSimFeatureVisualizerPlugin plugin; + private GraphType featureGraphType; + private GraphDisplayOptions featureGraphOptions; + private AttributedGraph featureGraph; + + /** + * Creates an action for drawing BSim feature graphs and highlighting relevant tokens in the + * decompiler. + * @param provider provider + * @param plugin plugin + */ + public HighlightAndGraphAction(BsfvTableProvider provider, BSimFeatureVisualizerPlugin plugin) { + super(NAME, plugin.getName()); + this.provider = provider; + this.plugin = plugin; + featureGraphType = new BSimFeatureGraphType(); + featureGraphOptions = + new BSimFeatureGraphDisplayOptions(featureGraphType, plugin.getTool()); + setPopupMenuData(new MenuData(new String[] { "Highlight and Graph" })); + setDescription("Create a graph and decompiler highlight for this BSim feature"); + HelpLocation help = new HelpLocation(plugin.getName(), "Visualizing_BSim_Features"); + setHelpLocation(help); + } + + @Override + public boolean isAddToPopup(ActionContext context) { + return true; + } + + @Override + public boolean isEnabledForContext(ActionContext context) { + return provider.getModel().getLastSelectedObjects().size() > 0; + } + + @Override + public void actionPerformed(ActionContext context) { + GraphDisplayBroker graphDisplayBroker = + plugin.getTool().getService(GraphDisplayBroker.class); + if (graphDisplayBroker == null) { + Msg.showError(this, plugin.getTool().getToolFrame(), "BSimFeatureVisualizer Error", + "No graph display providers found: Please add a graph display provider to " + + "your tool"); + return; + } + GhidraTable bsimFeatureTable = provider.getTable(); + BsfvTableModel model = provider.getModel(); + if (bsimFeatureTable.getSelectedRow() == -1) { + return; + } + PcodeOpAST pcode = model.getOpAt(bsimFeatureTable.getSelectedRow()); + PcodeOpAST previousPcode = model.getPreviousOpAt(bsimFeatureTable.getSelectedRow()); + featureGraph = new AttributedGraph("BSim Feature Graph", featureGraphType); + StringBuilder graphName = new StringBuilder(); + switch (model.getFeatureTypeAt(bsimFeatureTable.getSelectedRow())) { + case DATA_FLOW: + addDataFlowFeatureGraph(pcode, BSimFeatureGraphType.DATAFLOW_WINDOW_SIZE, + featureGraph, true); + graphName.append("DATA("); + graphName.append(pcode.getMnemonic()); + graphName.append(")@"); + graphName.append(pcode.getSeqnum().getTarget()); + break; + case CONTROL_FLOW: + int blockIndex = model.getBlockIndexAt(bsimFeatureTable.getSelectedRow()); + addControlFlowFeatureGraph(blockIndex, featureGraph); + graphName.append("CONTROL@"); + graphName.append(model.getBasicBlockStart(bsimFeatureTable.getSelectedRow())); + break; + case COMBINED: + blockIndex = model.getBlockIndexAt(bsimFeatureTable.getSelectedRow()); + addControlFlowFeatureGraph(blockIndex, featureGraph); + addDataFlowFeatureGraph(pcode, 1 + BSimFeatureGraphType.DATAFLOW_WINDOW_SIZE / 2, + featureGraph, true); + graphName.append("COMBINED("); + graphName.append(pcode.getMnemonic()); + graphName.append(")@"); + graphName.append(pcode.getSeqnum().getTarget()); + break; + case DUAL_FLOW: + addDataFlowFeatureGraph(pcode, 1 + BSimFeatureGraphType.DATAFLOW_WINDOW_SIZE / 2, + featureGraph, true); + addDataFlowFeatureGraph(previousPcode, + 1 + BSimFeatureGraphType.DATAFLOW_WINDOW_SIZE / 2, featureGraph, false); + graphName.append("DUAL("); + graphName.append(pcode.getMnemonic()); + graphName.append(","); + graphName.append(previousPcode.getMnemonic()); + graphName.append(")@"); + graphName.append(pcode.getSeqnum().getTarget()); + break; + case COPY_SIG: + blockIndex = model.getBlockIndexAt(bsimFeatureTable.getSelectedRow()); + addCopySigFeatureGraph(blockIndex, featureGraph); + graphName.append("COPY_SIG@"); + graphName.append(model.getBasicBlockStart(bsimFeatureTable.getSelectedRow())); + break; + default: + throw new IllegalArgumentException(); + } + GraphDisplay graphDisplay = null; + + try { + boolean reuseGraph = plugin.getReuseGraph(); + graphDisplay = graphDisplayBroker.getDefaultGraphDisplay(reuseGraph, TaskMonitor.DUMMY); + } + catch (GraphException e) { + Msg.showError(this, null, "Graph Error", e.getMessage(), e); + return; + } + + if (graphDisplay == null) { + Msg.showError(this, null, "null GraphDisplay", "null GraphDisplay"); + } + else { + try { + //just use a dummy TaskMonitor since these graphs are small + graphDisplay.setGraph(featureGraph, featureGraphOptions, graphName.toString(), + false, TaskMonitor.DUMMY); + graphDisplay.setGraphDisplayListener(new BsfvGraphDisplayListener(plugin.getTool(), + plugin.getCurrentProgram(), graphDisplay)); + } + catch (CancelledException e) { + return; + } + } + DecompilerHighlightService highlightService = plugin.getDecompilerHighlightService(); + if (highlightService == null) { + Msg.showError(this, null, "DecompilerHighlightService not found", + "DecompilerHighlightService not found"); + } + else { + if (!plugin.getHighlightByRow()) { + BsfvTokenHighlightMatcher tokenMatcher = new BsfvTokenHighlightMatcher( + model.getRowObject(bsimFeatureTable.getSelectedRow()), model.getHighFunction(), + plugin); + DecompilerHighlighter highlighter = + highlightService.createHighlighter(BSIM_FEATURE_HIGHLIGHTER_NAME, tokenMatcher); + highlighter.applyHighlights(); + } + } + + } + + /** + * Returns the {@link AttributedGraph} created by (the last firing of) this action. + * @return graph + */ + AttributedGraph getGraph() { + return featureGraph; + } + + private void addCopySigFeatureGraph(int blockIndex, AttributedGraph graph) { + //at the moment just graph one block + //future improvement: search block for standalone copies and graph each one + AttributedVertex baseBlockVertex = + graph.addVertex(BSimFeatureGraphType.COPY_PREFIX + Integer.toString(blockIndex)); + HighFunction hfunction = provider.getModel().getHighFunction(); + PcodeBlockBasic basicBlock = hfunction.getBasicBlocks().get(blockIndex); + setVertexTypeAndAttributes(baseBlockVertex, basicBlock, + BSimFeatureGraphType.COPY_SIGNATURE); + + } + + private void addControlFlowFeatureGraph(Integer baseBlockIndex, AttributedGraph graph) { + + Map indicesToVertices = new HashMap<>(); + HighFunction hfunction = provider.getModel().getHighFunction(); + ArrayList bBlocks = hfunction.getBasicBlocks(); + PcodeBlockBasic baseBlock = bBlocks.get(baseBlockIndex); + if (baseBlock.getStart() == null || baseBlock.getStop() == null) { + Msg.info(this, "null base block: baseBlockIndex " + baseBlockIndex); + return; + } + + AttributedVertex baseBlockVertex = graph.addVertex( + BSimFeatureGraphType.CONTROL_FLOW_PREFIX + Integer.toString(baseBlockIndex)); + setVertexTypeAndAttributes(baseBlockVertex, baseBlock, + BSimFeatureGraphType.BASE_BLOCK_VERTEX); + indicesToVertices.put(baseBlockIndex, baseBlockVertex); + + //add the parents + for (int i = 0, numParents = baseBlock.getInSize(); i < numParents; i++) { + PcodeBlock parentBlock = baseBlock.getIn(i); + AttributedVertex parentVertex = + indicesToVertices.computeIfAbsent(parentBlock.getIndex(), + x -> graph.addVertex(BSimFeatureGraphType.CONTROL_FLOW_PREFIX + + Integer.toString(parentBlock.getIndex()))); + setVertexTypeAndAttributes(parentVertex, parentBlock, + BSimFeatureGraphType.PARENT_BLOCK_VERTEX); + addGraphEdge(graph, parentBlock, parentVertex, baseBlock, baseBlockVertex); + + //add grandparents + for (int j = 0, numGrandParents = parentBlock.getInSize(); j < numGrandParents; j++) { + PcodeBlock grandParentBlock = parentBlock.getIn(j); + AttributedVertex grandParentVertex = + indicesToVertices.computeIfAbsent(grandParentBlock.getIndex(), + x -> graph.addVertex(BSimFeatureGraphType.CONTROL_FLOW_PREFIX + + Integer.toString(grandParentBlock.getIndex()))); + setVertexTypeAndAttributes(grandParentVertex, grandParentBlock, + BSimFeatureGraphType.GRANDPARENT_BLOCK_VERTEX); + addGraphEdge(graph, grandParentBlock, grandParentVertex, parentBlock, parentVertex); + } + //add the siblings + for (int j = 0, numSiblings = parentBlock.getOutSize(); j < numSiblings; j++) { + PcodeBlock siblingBlock = parentBlock.getOut(j); + if (siblingBlock.equals(baseBlock)) { + continue; + } + AttributedVertex siblingVertex = + indicesToVertices.computeIfAbsent(siblingBlock.getIndex(), + x -> graph.addVertex(BSimFeatureGraphType.CONTROL_FLOW_PREFIX + + Integer.toString(siblingBlock.getIndex()))); + setVertexTypeAndAttributes(siblingVertex, siblingBlock, + BSimFeatureGraphType.SIBLING_BLOCK_VERTEX); + addGraphEdge(graph, parentBlock, parentVertex, siblingBlock, siblingVertex); + } + } + //add the children + for (int i = 0, numChildren = baseBlock.getOutSize(); i < numChildren; i++) { + PcodeBlock childBlock = baseBlock.getOut(i); + AttributedVertex childVertex = indicesToVertices.computeIfAbsent(childBlock.getIndex(), + x -> graph.addVertex(BSimFeatureGraphType.CONTROL_FLOW_PREFIX + + Integer.toString(childBlock.getIndex()))); + setVertexTypeAndAttributes(childVertex, childBlock, + BSimFeatureGraphType.CHILD_BLOCK_VERTEX); + addGraphEdge(graph, baseBlock, baseBlockVertex, childBlock, childVertex); + } + } + + private void setVertexTypeAndAttributes(AttributedVertex vertex, PcodeBlock block, + String vertexType) { + String existingType = vertex.getVertexType(); + if (existingType == null) { + vertex.setVertexType(vertexType); + } + else { + if (!existingType.equals(vertexType) && + !existingType.equals(BSimFeatureGraphType.BASE_BLOCK_VERTEX)) { + vertex.setVertexType(BSimFeatureGraphType.BSIM_NEIGHBOR_VERTEX); + } + return; //attributes already set + } + vertex.setAttribute(BSimFeatureGraphType.BLOCK_START, block.getStart().toString()); + vertex.setAttribute(BSimFeatureGraphType.BLOCK_STOP, block.getStop().toString()); + vertex.setAttribute(BSimFeatureGraphType.CALL_STRING, + provider.getModel().getCallString(block.getIndex())); + + } + + private void addGraphEdge(AttributedGraph graph, PcodeBlock sourceBlock, + AttributedVertex sourceVertex, PcodeBlock targetBlock, AttributedVertex targetVertex) { + AttributedEdge edge = graph.addEdge(sourceVertex, targetVertex); + if (sourceBlock.getOutSize() != 2) { + edge.setEdgeType(BSimFeatureGraphType.CONTROL_FLOW_DEFAULT_EDGE); + return; + } + //sourceBlock must end in CBRANCH, true/false edge incorporated into hash + if (sourceBlock.getFalseOut().equals(targetBlock)) { + edge.setEdgeType(BSimFeatureGraphType.FALSE_EDGE); + } + else { + edge.setEdgeType(BSimFeatureGraphType.TRUE_EDGE); + } + return; + } + + //some paths in the dataflow graph (involving chains of COPY, INDIRECT, and MULTIEQUAL ops) are + //collapsed before signature generation. This leads to varnodes in the dataflow graph which + //have defining ops but which are not base varnodes of DATA_FLOW features. Rather than + ///re-implement the collapsing algorithm, we just keep track of which varnodes do not have + //features. + //The collapsed paths *are* shown in the BSim feature graphs created by this method, but are + //colored to indicate that they are collapsed during feature generation + private AttributedVertex addDataFlowFeatureGraph(PcodeOpAST pcode, int windowSize, + AttributedGraph graph, boolean primaryBase) { + Queue varnodes = new LinkedList<>(); + VarnodeAST vn = (VarnodeAST) pcode.getOutput(); + + Set featuredOps = provider.getModel().getFeaturedOps(); + Map varnodesToVertices = new HashMap<>(); + Map opsToVertices = new HashMap<>(); + AttributedVertex baseVertex = null; + if (vn == null) { + baseVertex = graph.addVertex( + BSimFeatureGraphType.DATAFLOW_PREFIX + Integer.toString(graph.getVertexCount()), + BSimFeatureGraphType.VOID_BASE); + vn = new VarnodeAST(Address.NO_ADDRESS, 0, 0); + } + else { + baseVertex = graph.addVertex( + BSimFeatureGraphType.DATAFLOW_PREFIX + Integer.toString(graph.getVertexCount()), + vn.toString(plugin.getCurrentProgram().getLanguage())); + baseVertex.setAttribute(BSimFeatureGraphType.SIZE, Integer.toString(vn.getSize())); + } + + baseVertex.setVertexType(primaryBase ? BSimFeatureGraphType.BASE_VARNODE_VERTEX + : BSimFeatureGraphType.SECONDARY_BASE_VARNODE_VERTEX); + varnodesToVertices.put(vn, baseVertex); + + DataflowQueueElement base = new DataflowQueueElement(vn, windowSize); + varnodes.add(base); + + //elements of queue should be correspond to varnodes (or artificial "base" varnode) which + //have a defining/corresponding pcode op + //so no constants or function inputs in the queue (but they will be added to the *graph*) + while (!varnodes.isEmpty()) { + DataflowQueueElement currentElement = varnodes.poll(); + VarnodeAST outputVarnode = currentElement.vn; + AttributedVertex varnodeVertex = varnodesToVertices.get(outputVarnode); + PcodeOpAST currentPcode = null; + if (outputVarnode.getAddress().equals(Address.NO_ADDRESS)) { + //this can only happen when the pcode argument to this method has no output + currentPcode = pcode; + } + else { + currentPcode = (PcodeOpAST) outputVarnode.getDef(); + } + //varnode vertex is collapsed if corresponding varnode is defined by a collapsed op + //don't collapse special case where op has no output + boolean collapsedOp = false; + if (!featuredOps.contains(currentPcode) && currentPcode != pcode) { + collapsedOp = true; + } + if (collapsedOp) { + varnodeVertex.setVertexType(BSimFeatureGraphType.COLLAPSED_VARNODE); + if (opsToVertices.containsKey(currentPcode)) { + continue; //avoid getting stuck in collapsed loop + } + } + AttributedVertex pcodeVertex = opsToVertices.computeIfAbsent(currentPcode, + x -> graph.addVertex( + BSimFeatureGraphType.DATAFLOW_PREFIX + Integer.toString(graph.getVertexCount()), + x.getMnemonic())); + if (collapsedOp) { + pcodeVertex.setVertexType(BSimFeatureGraphType.COLLAPSED_OP); + } + else { + pcodeVertex.setVertexType(BSimFeatureGraphType.PCODE_OP_VERTEX); + } + pcodeVertex.setAttribute(BSimFeatureGraphType.OP_ADDRESS, + currentPcode.getSeqnum().getTarget().toString()); + pcodeVertex.setAttribute(BSimFeatureGraphType.PCODE_OUTPUT, + currentPcode.getOutput() == null ? BSimFeatureGraphType.VOID_BASE + : currentPcode.getOutput() + .toString(plugin.getCurrentProgram().getLanguage())); + AttributedEdge edge = graph.addEdge(pcodeVertex, varnodeVertex); + if (collapsedOp) { + edge.setEdgeType(BSimFeatureGraphType.COLLAPSED_OUT); + } + else { + edge.setEdgeType(BSimFeatureGraphType.DATAFLOW_OUT); + } + int start = 0; + int stop = currentPcode.getNumInputs(); + switch (currentPcode.getOpcode()) { + case PcodeOp.CPOOLREF: + stop = 1; + break; + case PcodeOp.INDIRECT: + stop -= 1; + break; + case PcodeOp.CALL: + case PcodeOp.CALLIND: + case PcodeOp.CALLOTHER: + case PcodeOp.CBRANCH: //should only occur with COMBINED or DUAL_FLOW features + case PcodeOp.LOAD: + case PcodeOp.RETURN: + case PcodeOp.STORE: + start += 1; + break; + case PcodeOp.INT_LEFT: + case PcodeOp.INT_RIGHT: + case PcodeOp.INT_SRIGHT: + case PcodeOp.SUBPIECE: + if (currentPcode.getInput(1).isConstant()) { + stop -= 1; + } + break; + default: + break; + } + for (int j = start; j < stop; j++) { + Varnode iv = currentPcode.getInput(j); + if (iv == null) { + Msg.info(this, "Null input for pcode " + currentPcode.getMnemonic()); + continue; + } + VarnodeAST inputVarnode = (VarnodeAST) iv; + AttributedVertex inputVarnodeVertex = + varnodesToVertices.computeIfAbsent(inputVarnode, + x -> graph.addVertex( + BSimFeatureGraphType.DATAFLOW_PREFIX + + Integer.toString(graph.getVertexCount()), + inputVarnode.toString(plugin.getCurrentProgram().getLanguage()))); + inputVarnodeVertex.setVertexType(BSimFeatureGraphType.DEFAULT_VERTEX); + if (inputVarnode.isConstant() && inputVarnode.isInput()) { + inputVarnodeVertex.setVertexType(BSimFeatureGraphType.CONSTANT_FUNCTION_INPUT); + } + else { + if (inputVarnode.isConstant()) { + inputVarnodeVertex.setVertexType(BSimFeatureGraphType.CONSTANT_VERTEX); + } + if (inputVarnode.isInput()) { + inputVarnodeVertex.setVertexType(BSimFeatureGraphType.FUNCTION_INPUT); + } + } + if (inputVarnode.isAddress()) { + inputVarnodeVertex.setVertexType(BSimFeatureGraphType.VARNODE_ADDRESS); + } + inputVarnodeVertex.setAttribute(BSimFeatureGraphType.SIZE, + Integer.toString(inputVarnode.getSize())); + edge = graph.addEdge(inputVarnodeVertex, pcodeVertex); + if (collapsedOp) { + edge.setEdgeType(BSimFeatureGraphType.COLLAPSED_IN); + } + else { + edge.setEdgeType(BSimFeatureGraphType.DATAFLOW_IN); + } + if (inputVarnode.getDef() != null) { + int numHops = collapsedOp ? currentElement.remainingHops + : currentElement.remainingHops - 1; + if (numHops > 0) { + DataflowQueueElement inputElement = + new DataflowQueueElement(inputVarnode, numHops); + varnodes.add(inputElement); + } + } + } + } + return baseVertex; + } + + private class DataflowQueueElement { + public int remainingHops; + public VarnodeAST vn; + + public DataflowQueueElement(VarnodeAST vn, int remainingHops) { + this.vn = vn; + this.remainingHops = remainingHops; + } + } + +} diff --git a/Ghidra/Features/Base/src/main/help/help/topics/FunctionComparison/FunctionComparison.htm b/Ghidra/Features/Base/src/main/help/help/topics/FunctionComparison/FunctionComparison.htm index 1d846c6c81..54c43fb8de 100644 --- a/Ghidra/Features/Base/src/main/help/help/topics/FunctionComparison/FunctionComparison.htm +++ b/Ghidra/Features/Base/src/main/help/help/topics/FunctionComparison/FunctionComparison.htm @@ -15,8 +15,8 @@ functions in a simple side-by-side panel.

To begin, select a function (or multiple functions) from the listing or - the function table. Then right-click and select the Compare Selected - Functions option.

+ the function table. + Then right-click and select the Compare Selected Functions option.

A new function comparison window will appear (subsequent invocations of this option will create a new tab in the existing window).

@@ -33,9 +33,9 @@

The Listing View shows a pair of listings side by side so the functions can be compared. The highlights in the two listings indicate where there are possible differences in the bytes and instructions between the functions. The background highlights that have a - default color of gray are where bytes or instructions differ - wherever the instructions could be matched up between the two functions. The code units with - white background also have a matching code unit in the other function but do not have any + default color of gray are where bytes or instructions + differ wherever the instructions could be matched up between the two functions. The code units + with white background also have a matching code unit in the other function but do not have any differences based on the current difference settings. Any instructions that couldn't be automatically matched up are highlighted with a default background color of blue. @@ -378,7 +378,8 @@

Listing Code Comparison Options

-

These are the tool options that can be adjusted for the dual listing view of a Function Comparison.

+

These are the tool options that can be adjusted for the dual listing view of a Function + Comparison.


@@ -405,6 +406,98 @@
+

Decompiler Diff View

+ +
+

The Decompiler Diff View shows a pair of decompilers side by side. + Individual tokens can be highlighted in various colors to show differences and matches between + the two functions. The default colors are described in the following table; users can change + these colors from the Properties action in the popup menu of the Decompiler Diff View + or from the Decompiler Code Comparison entry in the Tool Options.

+

+ +

 Decompiler Code Comparison Options +

+
+
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
IndicatorDefault ColorDescription
Differing Tokens  Highlights (eligible) tokens in each decompiled views which do not have matches + in the other function.
Ineligible Tokens  This color is used to highlight a focused token when it is not eligible for a + match via the matching algorithm. For example, whitespace tokens and variable + declarations are never assigned matches.
Matched Tokens  If the focused token has a match in the other function, all tokens involved in + the match are highlighted in this color.
Unmatched Tokens  This color is used to highlight the focused token when it is eligible for a match + but does not have one.
+ +
+
+
+
+
+
+ +

Synchronized scrolling works by matching a line in one decompiled function with a line in + the other decompiled function. The two functions are aligned relative to the matched + line; scrolling up or down will then move each decompiled function by the same amount. + Whenever you click on a token, the functions are re-aligned. If the token you clicked on + has a match, the functions will be re-aligned using the corresponding line. If the token + does not have a match, a search is performed for the nearest token in the same function + which does have a match, and the functions are re-aligned using the search result. +

+ + +

Decompiler Code Comparison Actions

+ +

Compare Matching Callees

+
+

This action is available on matched tokens corresponding to function calls. It will open + a new function comparison window populated with the called functions.

+
+ +

Show Decompilers Side-by-Side

+
+

This toggles the decompiler panels between a vertical split and a horizontal split.

+
+
+

Comparing Multiple Functions

@@ -415,7 +508,8 @@

The following toolbar options are available:

-

Add To Existing Comparison

+

+ Add To Existing Comparison

Allows the user to add functions to the current comparison window. When selected, the following table containing all functions in the current program is displayed:

@@ -427,17 +521,21 @@ sides of the comparison window.

-

 Remove Function From Comparison

+

+  Remove Function From Comparison

Removes the function in the focused panel from the comparison. This will remove the function from both the source and target selection pulldowns.

-

 Go To Next Function

+

+  Go To Next Function

Navigates to the next available function in the selection pulldown

-

 Go To Previous Function

+

+  Go To Previous Function

Navigates to the previous available function in the selection pulldown

-

 Navigate To Selected Function

+

+  Navigate To Selected Function

When toggled on, the function comparison panels become navigable, meaning a mouse click on the panel or change in the function being viewed will result in a GoTo event being generated. This allows other panels @@ -448,13 +546,6 @@ pink border.

-

Decompiler Code Comparison Options

- -

Show Decompiler Side-by-Side

-
-

This toggles the decompiler panels between a vertical split and a horizontal split. -

-

Other Function Comparison Actions

The following are additional actions that are available in the Function Comparison @@ -503,35 +594,27 @@

-

Apply Function Signature To Other - Side

+

Apply Actions

+

These actions apply information from the other function to this function. Currently, + the following apply actions are provided:

+ -
-

The Apply Function Signature To Other Side action is available in the Function - Comparison window from the popup menu. This will apply the function signature from the - side of the panel that has focus (the one with the pink border around it) to the function - in the other side.

- -

When you apply the signature it updates the other side's function so its signature - will match the function's signature in the side with focus as closely as possible. This - will try to create conflict names if necessary for the function and its parameters.

- -

To apply the signature from one function to the other:

- -
    -
  1. Right mouse click in the side of the Function Comparison Window that has the - function signature that you want to apply.
    2. - -
  2. Choose the Apply Function Signature To Other Side option from the menu.
    3. -
- -

The function signature in the non-focused side of the Function Comparison window - should update if the apply succeeded.

-
- -
- -

Provided By:  FunctionComparisonPlugin

+

Provided By:  FunctionComparisonPlugin

Related Topics:

diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/cmd/function/ApplyFunctionSignatureCmd.java b/Ghidra/Features/Base/src/main/java/ghidra/app/cmd/function/ApplyFunctionSignatureCmd.java index b8b90636af..c254246d6d 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/cmd/function/ApplyFunctionSignatureCmd.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/cmd/function/ApplyFunctionSignatureCmd.java @@ -28,6 +28,7 @@ import ghidra.program.model.listing.*; import ghidra.program.model.listing.Function.FunctionUpdateType; import ghidra.program.model.pcode.Varnode; import ghidra.program.model.symbol.*; +import ghidra.program.util.DataTypeCleaner; import ghidra.util.Msg; import ghidra.util.exception.DuplicateNameException; import ghidra.util.exception.InvalidInputException; @@ -43,59 +44,120 @@ public class ApplyFunctionSignatureCmd extends BackgroundCommand { private SourceType source; private FunctionRenameOption functionRenameOption; private boolean preserveCallingConvention; + private boolean applyEmptyComposites; + private DataTypeConflictHandler conflictHandler; private FunctionSignature signature; private Program program; /** - * Constructs a new command for creating a function. + * Constructs a new command for applying a signature to an existing function. + *
* Only a function with a default name will be renamed to the function signature's name * (see {@link FunctionRenameOption#RENAME_IF_DEFAULT}). - * @param entry entry point address for the function to be created. + *
+ * All datatypes will be resolved using the + * {@link DataTypeConflictHandler#DEFAULT_HANDLER default conflict handler}. + * + * @param entry entry point address for the function to be created. * @param signature function signature to apply - * @param source the source of this function signature + * @param source the source of this function signature */ public ApplyFunctionSignatureCmd(Address entry, FunctionSignature signature, SourceType source) { - this(entry, signature, source, false, FunctionRenameOption.RENAME_IF_DEFAULT); + this(entry, signature, source, false, false, DataTypeConflictHandler.DEFAULT_HANDLER, + FunctionRenameOption.RENAME_IF_DEFAULT); } /** - * Constructs a new command for creating a function. - * @param entry entry point address for the function to be created. + * Constructs a new command for applying a signature to an existing function. + *
+ * All datatypes will be resolved using the + * {@link DataTypeConflictHandler#DEFAULT_HANDLER default conflict handler}. + * + * @param entry entry point address for the function to be created. * @param signature function signature to apply - * @param source the source of this function signature + * @param source the source of this function signature * @param preserveCallingConvention if true the function calling convention will not be changed * @param forceSetName true if name of the function should be set to the name, otherwise name - * will only be set name if currently default (e.g., FUN_1234). A value of true is equivalent to - * {@link FunctionRenameOption#RENAME}, while a value of false is equivalent to - * {@link FunctionRenameOption#RENAME_IF_DEFAULT}. + * will only be set name if currently default (e.g., FUN_1234). A value of + * true is equivalent to {@link FunctionRenameOption#RENAME}, while a value + * of false is equivalent to {@link FunctionRenameOption#RENAME_IF_DEFAULT}. */ @Deprecated(since = "10.3", forRemoval = true) public ApplyFunctionSignatureCmd(Address entry, FunctionSignature signature, SourceType source, boolean preserveCallingConvention, boolean forceSetName) { - this(entry, signature, source, preserveCallingConvention, + this(entry, signature, source, preserveCallingConvention, false, + DataTypeConflictHandler.DEFAULT_HANDLER, forceSetName ? FunctionRenameOption.RENAME : FunctionRenameOption.RENAME_IF_DEFAULT); } /** - * Constructs a new command for creating a function. - * @param entry entry point address for the function to be created. + * Constructs a new command for applying a signature to an existing function. + *
+ * All datatypes will be resolved using the + * {@link DataTypeConflictHandler#DEFAULT_HANDLER default conflict handler}. + * + * @param entry entry point address for the function to be created. * @param signature function signature to apply - * @param source the source of this function signature + * @param source the source of this function signature * @param preserveCallingConvention if true the function calling convention will not be changed * @param functionRenameOption controls renaming of the function using the name from the - * specified function signature. + * specified function signature. */ + @Deprecated(since = "11.0", forRemoval = true) public ApplyFunctionSignatureCmd(Address entry, FunctionSignature signature, SourceType source, boolean preserveCallingConvention, FunctionRenameOption functionRenameOption) { + this(entry, signature, source, preserveCallingConvention, false, + DataTypeConflictHandler.DEFAULT_HANDLER, functionRenameOption); + } + + /** + * Constructs a new command for applying a signature to an existing function. + * + * @param entry entry point address for the function to be created. + * @param signature function signature to apply + * @param source the source of this function signature + * @param preserveCallingConvention if true the function calling convention will not be changed + * @param applyEmptyComposites If true, applied composites will be resolved without their + * respective components if the type does not already exist in the + * destination datatype manager. If false, normal type resolution + * will occur. + * @param conflictHandler conflict handler to be used when applying datatypes to the + * destination program. If this value is not null or + * {@link DataTypeConflictHandler#DEFAULT_HANDLER} the datatypes will be + * resolved prior to updating the destinationFunction. This handler + * will provide some control over how applied datatype are handled when + * they conflict with existing datatypes. + * See {@link DataTypeConflictHandler} which provides some predefined + * handlers. + * @param functionRenameOption controls renaming of the function using the name from the + * specified function signature. + */ + public ApplyFunctionSignatureCmd(Address entry, FunctionSignature signature, SourceType source, + boolean preserveCallingConvention, boolean applyEmptyComposites, + DataTypeConflictHandler conflictHandler, FunctionRenameOption functionRenameOption) { super("Create Function", true, false, false); this.entryPt = entry; this.signature = signature; this.source = source; this.preserveCallingConvention = preserveCallingConvention; + this.applyEmptyComposites = applyEmptyComposites; + this.conflictHandler = + (conflictHandler == null) ? DataTypeConflictHandler.DEFAULT_HANDLER : conflictHandler; this.functionRenameOption = functionRenameOption; } + private DataType prepareDataType(DataType dt, DataTypeManager destinationDtm, + DataTypeCleaner dtCleaner) { + if (dtCleaner != null) { + dt = dtCleaner.clean(dt); + } + if (conflictHandler != DataTypeConflictHandler.DEFAULT_HANDLER) { + dt = destinationDtm.resolve(dt, conflictHandler); + } + return dt; + } + @Override public boolean applyTo(DomainObject obj, TaskMonitor monitor) { program = (Program) obj; @@ -137,22 +199,32 @@ public class ApplyFunctionSignatureCmd extends BackgroundCommand { CompilerSpec compilerSpec = program.getCompilerSpec(); String conventionName = getCallingConvention(func, compilerSpec); - + DataType returnDt = signature.getReturnType(); ParameterDefinition[] args = signature.getArguments(); - List params = createParameters(compilerSpec, conventionName, args); - SymbolTable symbolTable = program.getSymbolTable(); + ProgramBasedDataTypeManager targetDtm = func.getProgram().getDataTypeManager(); + DataTypeCleaner dtCleaner = + applyEmptyComposites ? new DataTypeCleaner(targetDtm, true) : null; try { + if (dtCleaner != null || conflictHandler != DataTypeConflictHandler.DEFAULT_HANDLER) { + for (ParameterDefinition arg : args) { + arg.setDataType(prepareDataType(arg.getDataType(), targetDtm, dtCleaner)); + } + returnDt = prepareDataType(returnDt, targetDtm, dtCleaner); + } + + List params = createParameters(compilerSpec, conventionName, args); + + SymbolTable symbolTable = program.getSymbolTable(); adjustParameterNamesToAvoidConflicts(symbolTable, func, params); - ReturnParameterImpl returnParam = - new ReturnParameterImpl(signature.getReturnType(), program); + ReturnParameterImpl returnParam = new ReturnParameterImpl(returnDt, program); func.updateFunction(conventionName, returnParam, params, FunctionUpdateType.DYNAMIC_STORAGE_FORMAL_PARAMS, false, source); func.setVarArgs(signature.hasVarArgs()); - + // Only apply noreturn if signature has it set if (signature.hasNoReturn()) { func.setNoReturn(signature.hasNoReturn()); @@ -163,6 +235,11 @@ public class ApplyFunctionSignatureCmd extends BackgroundCommand { throw new InvalidInputException( "Parameter name conflict, likely due to concurrent operation"); } + finally { + if (dtCleaner != null) { + dtCleaner.close(); + } + } updateStackPurgeSize(func, program); @@ -198,8 +275,7 @@ public class ApplyFunctionSignatureCmd extends BackgroundCommand { private void adjustParameterNamesToAvoidConflicts(SymbolTable symbolTable, Function function, List params) throws DuplicateNameException, InvalidInputException { - for (int i = 0; i < params.size(); i++) { - Parameter param = params.get(i); + for (Parameter param : params) { String name = param.getName(); if (name == null || SymbolUtilities.isDefaultParameterName(name)) { continue; diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/analysis/ConstantPropagationContextEvaluator.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/analysis/ConstantPropagationContextEvaluator.java index 088b4de548..d3e345d530 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/analysis/ConstantPropagationContextEvaluator.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/analysis/ConstantPropagationContextEvaluator.java @@ -56,7 +56,7 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter private long minStoreLoadOffset = 4; private long minSpeculativeOffset = 1024; // from the beginning of memory private long maxSpeculativeOffset = 256; // from the end of memory - + protected TaskMonitor monitor; private final int NULL_TERMINATOR_PROBE = -1; @@ -73,22 +73,23 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter this.trustMemoryWrite = trustMemoryWrite; } - public ConstantPropagationContextEvaluator(TaskMonitor monitor, - boolean trustWriteMemOption, long minStoreLoadRefAddress, - long minSpeculativeRefAddress, long maxSpeculativeRefAddress) { + public ConstantPropagationContextEvaluator(TaskMonitor monitor, boolean trustWriteMemOption, + long minStoreLoadRefAddress, long minSpeculativeRefAddress, + long maxSpeculativeRefAddress) { this(monitor, trustWriteMemOption); this.minStoreLoadOffset = minStoreLoadRefAddress; this.minSpeculativeOffset = minSpeculativeRefAddress; this.maxSpeculativeOffset = maxSpeculativeRefAddress; } - + /** * Set option to trust reads from memory that is marked writeable * * @param trustWriteableMemOption true to trust values read from memory that is marked writable * @return this */ - public ConstantPropagationContextEvaluator setTrustWritableMemory(boolean trustWriteableMemOption) { + public ConstantPropagationContextEvaluator setTrustWritableMemory( + boolean trustWriteableMemOption) { trustMemoryWrite = trustWriteableMemOption; return this; } @@ -99,22 +100,24 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter * @param minSpeculativeRefAddress minimum address offset * @return this */ - public ConstantPropagationContextEvaluator setMinSpeculativeOffset(long minSpeculativeRefAddress) { + public ConstantPropagationContextEvaluator setMinSpeculativeOffset( + long minSpeculativeRefAddress) { minSpeculativeOffset = minSpeculativeRefAddress; return this; } - + /** * Set maximum speculative memory offset for references * * @param maxSpeculativeRefAddress maximum address offset * @return this */ - public ConstantPropagationContextEvaluator setMaxSpeculativeOffset(long maxSpeculativeRefAddress) { + public ConstantPropagationContextEvaluator setMaxSpeculativeOffset( + long maxSpeculativeRefAddress) { maxSpeculativeOffset = maxSpeculativeRefAddress; return this; } - + /** * Set maximum speculative memory offset for references * @@ -125,18 +128,19 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter maxSpeculativeOffset = minStoreLoadRefAddress; return this; } - + /** * Set option to create complex data for pointers if the datatype is known * * @param doCreateData true to create complex data types if the data type is known * @return this */ - public ConstantPropagationContextEvaluator setCreateComplexDataFromPointers(boolean doCreateData) { + public ConstantPropagationContextEvaluator setCreateComplexDataFromPointers( + boolean doCreateData) { createDataFromPointers = doCreateData; return this; } - + /** * The computed destination set is useful if follow on switch analysis is to be done. * @@ -181,7 +185,7 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter @Override public boolean evaluateReference(VarnodeContext context, Instruction instr, int pcodeop, Address address, int size, DataType dataType, RefType refType) { - + // special check for parameters, evaluating the call, an uncomputed call wouldn't get here normally // really there should be another callback when adding parameters if (refType.isCall() && !refType.isComputed() && pcodeop == PcodeOp.UNIMPLEMENTED) { @@ -211,8 +215,8 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter } Program program = instr.getProgram(); - AutoAnalysisManager aMgr= AutoAnalysisManager.getAnalysisManager(program); - + AutoAnalysisManager aMgr = AutoAnalysisManager.getAnalysisManager(program); + // if data reference, handle data if (refType.isData()) { createPointedToData(aMgr, program, address, refType, dataType, size); @@ -221,8 +225,8 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter // if flowing to an address, disassemble it // only disassemble in executable memory Memory memory = program.getMemory(); - if (refType.isFlow() && !refType.isIndirect() && - !memory.isExternalBlockAddress(address) && memory.getExecuteSet().contains(address)) { + if (refType.isFlow() && !refType.isIndirect() && !memory.isExternalBlockAddress(address) && + memory.getExecuteSet().contains(address)) { Data udata = program.getListing().getUndefinedDataAt(address); if (udata != null) { DisassembleCommand cmd = new DisassembleCommand(address, null, true); @@ -238,39 +242,41 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter return true; } - - private int createPointedToData(AutoAnalysisManager aMgr, Program program, Address address, RefType refType, DataType dataType, int size) { + + private int createPointedToData(AutoAnalysisManager aMgr, Program program, Address address, + RefType refType, DataType dataType, int size) { // don't do in external memory if (program.getMemory().isExternalBlockAddress(address) || address.isExternalAddress()) { return 0; } - + // don't create if not in real memory if (!program.getMemory().contains(address)) { return 0; } - + // Get the data type that is pointed to, strip off pointer, or pointer typedef DataType pointedToDT = null; if (dataType instanceof Pointer ptr) { pointedToDT = ptr.getDataType(); } - else if ((dataType instanceof TypeDef typeDef && typeDef.getBaseDataType() instanceof Pointer ptr)) { + else if ((dataType instanceof TypeDef typeDef && + typeDef.getBaseDataType() instanceof Pointer ptr)) { pointedToDT = ptr.getDataType(); } - + // if this is a function pointer, create the code/function/signature if (pointedToDT instanceof FunctionDefinition funcDefn) { createFunctionApplySignature(aMgr, program, address, funcDefn); return dataType.getLength(); } - + // otherwise it is data - + if (dataType != null) { // System.out.println("@ " + address + " Data Type - " + dataType); } - + return createData(program, address, pointedToDT, size); } @@ -289,10 +295,10 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter if (data == null || !Undefined.isUndefined(data.getDataType())) { return 0; } - + // get a default undefined data type of the right size for the access DataType dt = Undefined.getUndefinedDataType(size); - + int maxLen = -1; if (createDataFromPointers && dataType != null) { DataType originalDT = dataType; @@ -301,35 +307,43 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter if (dataType instanceof TypeDef typeDef) { dataType = typeDef.getBaseDataType(); } - + if (dataType instanceof CharDataType) { // Use Terminated in case strings are referenced offcut - maxLen = getRestrictedStringLen(program, address, TerminatedStringDataType.dataType, MAX_CHAR_STRING__LEN); + maxLen = getRestrictedStringLen(program, address, TerminatedStringDataType.dataType, + MAX_CHAR_STRING__LEN); if (maxLen > 0) { dt = TerminatedStringDataType.dataType; } - } else if (dataType instanceof WideCharDataType) { - maxLen = getRestrictedStringLen(program, address, TerminatedUnicodeDataType.dataType, MAX_UNICODE_STRING_LEN); + } + else if (dataType instanceof WideCharDataType) { + maxLen = getRestrictedStringLen(program, address, + TerminatedUnicodeDataType.dataType, MAX_UNICODE_STRING_LEN); if (maxLen > 0) { dt = TerminatedUnicodeDataType.dataType; } - } else if (dataType instanceof Composite comp) { + } + else if (dataType instanceof Composite comp) { // create empty structures, they can get filled out later // if they don't fit later because they grow, then there will be an error at the location dt = originalDT; // original might have been a typedef, use original - } else if (dataType instanceof VoidDataType) { + } + else if (dataType instanceof VoidDataType) { // ptr to void should be ignored return 0; - } else if (dataType instanceof AbstractFloatDataType) { + } + else if (dataType instanceof AbstractFloatDataType) { dt = dataType; - } else { + } + else { // don't do any other types other than above for now return 0; } - } else if (size < 1 || size > 8) { + } + else if (size < 1 || size > 8) { return 0; } - + try { // create data at the location so that we record the access size // the data is undefined, and SHOULD be overwritten if something @@ -337,7 +351,8 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter if (maxLen > 0) { data = DataUtilities.createData(program, address, dt, maxLen, ClearDataMode.CLEAR_ALL_UNDEFINED_CONFLICT_DATA); - } else { + } + else { data = DataUtilities.createData(program, address, dt, -1, ClearDataMode.CLEAR_ALL_UNDEFINED_CONFLICT_DATA); } @@ -364,13 +379,13 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter // System.out.println("@ " + address + " - Typedef Function " + ((FunctionDefinition)ptrToDT).getPrototypeString()); Listing listing = program.getListing(); - + // defined data (that isn't an undefined) don't do anything Data data = listing.getDefinedDataContaining(address); if (data != null) { return; } - + // if memory is undefined bytes, don't disassemble or create function MemoryBlock block = program.getMemory().getBlock(address); if (block == null || !block.isInitialized()) { @@ -378,20 +393,22 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter } // normalize the address to where code should start (ARM and MIPS can be offset by one) - Address normalizedAddr = PseudoDisassembler.getNormalizedDisassemblyAddress(program, address); + Address normalizedAddr = + PseudoDisassembler.getNormalizedDisassemblyAddress(program, address); if (!listing.isUndefined(normalizedAddr, normalizedAddr)) { // if not undefined, must be an instruction to continue Instruction instr = listing.getInstructionAt(normalizedAddr); if (instr == null) { return; } - } else { + } + else { // if nothing defined here, disassemble address = PseudoDisassembler.setTargetContextForDisassembly(program, address); DisassembleCommand cmd = new DisassembleCommand(address, null, true); cmd.applyTo(program, monitor); } - + // see if there is an existing function FunctionManager funcMgr = program.getFunctionManager(); Function func = funcMgr.getFunctionAt(address); @@ -403,7 +420,7 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter return; } } - + if (func != null) { if (func.isThunk()) { return; @@ -413,29 +430,34 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter if (SourceType.ANALYSIS.isLowerPriorityThan(mostTrusted)) { return; } - } else { + } + else { CreateFunctionCmd createFunctionCmd = new CreateFunctionCmd(address, false); aMgr.schedule(createFunctionCmd, AnalysisPriority.FUNCTION_ANALYSIS.priority()); } - + // only apply the signature if actually creating data, the function/code has already been created if (!createDataFromPointers) { return; } - + // if no arguments, could be an opaque function pointer, don't apply arguments ParameterDefinition[] arguments = funcDefn.getArguments(); DataType returnType = funcDefn.getReturnType(); - if (arguments == null || (arguments.length == 0 && (returnType==null || Undefined.isUndefined(returnType)))) { + if (arguments == null || + (arguments.length == 0 && (returnType == null || Undefined.isUndefined(returnType)))) { return; } // applying function signatures considered data // don't change name, even default name - ApplyFunctionSignatureCmd applyFunctionSignatureCmd = new ApplyFunctionSignatureCmd(address, funcDefn, SourceType.ANALYSIS, true, FunctionRenameOption.NO_CHANGE); - aMgr.schedule(applyFunctionSignatureCmd, AnalysisPriority.FUNCTION_ANALYSIS.after().priority()); + ApplyFunctionSignatureCmd applyFunctionSignatureCmd = + new ApplyFunctionSignatureCmd(address, funcDefn, SourceType.ANALYSIS, true, false, + DataTypeConflictHandler.DEFAULT_HANDLER, FunctionRenameOption.NO_CHANGE); + aMgr.schedule(applyFunctionSignatureCmd, + AnalysisPriority.FUNCTION_ANALYSIS.after().priority()); } - + private SourceType getMostTrustedParameterSource(Function func) { SourceType highestSource = func.getSignatureSource(); Parameter[] parameters = func.getParameters(); @@ -476,7 +498,7 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter public boolean allowAccess(VarnodeContext context, Address addr) { return trustMemoryWrite; } - + /** * Looks at bytes at given address and converts to format String * @@ -484,28 +506,29 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter * @param pointer Pointer "type" of string * @return format String */ - int getRestrictedStringLen(Program program, Address address, AbstractStringDataType dataType, int maxLength) { - + int getRestrictedStringLen(Program program, Address address, AbstractStringDataType dataType, + int maxLength) { + maxLength = getMaxStringLength(program, address, maxLength); - MemoryBufferImpl memoryBuffer = - new MemoryBufferImpl(program.getMemory(), address); - - StringDataInstance stringDataInstance = dataType.getStringDataInstance(memoryBuffer, dataType.getDefaultSettings(), -1); + MemoryBufferImpl memoryBuffer = new MemoryBufferImpl(program.getMemory(), address); + + StringDataInstance stringDataInstance = + dataType.getStringDataInstance(memoryBuffer, dataType.getDefaultSettings(), -1); stringDataInstance.getStringDataTypeGuess(); - + int detectedLength = stringDataInstance.getStringLength(); if (detectedLength == -1) { return 0; } - + if (detectedLength > maxLength) { detectedLength = maxLength; } - + return detectedLength; } - + /** * Get the number of bytes to the next reference, or the max length * @@ -514,12 +537,13 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter * @return maximum length to create the string */ private int getMaxStringLength(Program program, Address address, int maxLen) { - AddressIterator refIter = program.getReferenceManager().getReferenceDestinationIterator(address.next(), true); + AddressIterator refIter = + program.getReferenceManager().getReferenceDestinationIterator(address.next(), true); Address next = refIter.next(); if (next == null) { return -1; } - + long len = -1; try { len = next.subtract(address); @@ -527,7 +551,8 @@ public class ConstantPropagationContextEvaluator extends ContextEvaluatorAdapter len = maxLen; } return (int) len; - } catch (IllegalArgumentException exc) { + } + catch (IllegalArgumentException exc) { // bad address subtraction } return (int) len; diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/function/EditFunctionSignatureDialog.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/function/EditFunctionSignatureDialog.java index 416059cbc5..ee87361506 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/function/EditFunctionSignatureDialog.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/function/EditFunctionSignatureDialog.java @@ -24,8 +24,7 @@ import ghidra.framework.cmd.Command; import ghidra.framework.cmd.CompoundCmd; import ghidra.framework.model.DomainObject; import ghidra.framework.plugintool.PluginTool; -import ghidra.program.model.data.DataTypeManager; -import ghidra.program.model.data.FunctionDefinitionDataType; +import ghidra.program.model.data.*; import ghidra.program.model.listing.Function; import ghidra.program.model.listing.FunctionSignature; import ghidra.program.model.symbol.SourceType; @@ -176,7 +175,8 @@ public class EditFunctionSignatureDialog extends AbstractEditFunctionSignatureDi return null; } cmd = new ApplyFunctionSignatureCmd(function.getEntryPoint(), definition, - SourceType.USER_DEFINED, true, FunctionRenameOption.RENAME); + SourceType.USER_DEFINED, true, false, DataTypeConflictHandler.DEFAULT_HANDLER, + FunctionRenameOption.RENAME); } CompoundCmd compoundCommand = new CompoundCmd("Update Function Signature"); diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparison.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparison.java index 8743fee1b0..dfb377a62b 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparison.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparison.java @@ -33,7 +33,8 @@ public class FunctionComparison implements Comparable { private Function source; /** Use a tree so functions are always kept in sorted order */ - private Set targets = new TreeSet<>(new FunctionComparator()); + private FunctionComparator functionComparator = new FunctionComparator(); + private Set targets = new TreeSet<>(functionComparator); /** * Returns the source function @@ -105,28 +106,7 @@ public class FunctionComparison implements Comparable { */ @Override public int compareTo(FunctionComparison o) { - if (o == null) { - return 1; - } - - String sourcePath = getSource().getProgram().getDomainFile().getPathname(); - String otherPath = o.getSource().getProgram().getDomainFile().getPathname(); - - String sourceName = getSource().getName(); - String otherName = o.getSource().getName(); - int result = sourcePath.compareTo(otherPath); - if (result != 0) { - return result; - } - - // equal paths - result = sourceName.compareTo(otherName); - if (result != 0) { - return result; - } - - // equal names - return getSource().getEntryPoint().compareTo(o.getSource().getEntryPoint()); + return functionComparator.compare(source, o.source); } /** @@ -144,21 +124,18 @@ public class FunctionComparison implements Comparable { String o1Path = o1.getProgram().getDomainFile().getPathname(); String o2Path = o2.getProgram().getDomainFile().getPathname(); - int result = o1Path.compareTo(o2Path); - if (result != 0) { - return result; - } - // equal paths String o1Name = o1.getName(); String o2Name = o2.getName(); - result = o1Name.compareTo(o2Name); - if (result != 0) { - return result; + + if (o1Path.equals(o2Path)) { + if (o1Name.equals(o2Name)) { + return o1.getEntryPoint().compareTo(o2.getEntryPoint()); + } + return o1Name.compareTo(o2Name); } - // equal names - return o1.getEntryPoint().compareTo(o2.getEntryPoint()); + return o1Path.compareTo(o2Path); } } } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPanel.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPanel.java index 59b0d2f2f1..7ddc827b98 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPanel.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPanel.java @@ -688,43 +688,16 @@ public class FunctionComparisonPanel extends JPanel implements ChangeListener { */ private List> getCodeComparisonPanels() { if (codeComparisonPanels == null) { - codeComparisonPanels = new ArrayList<>(); - Set> instances = - createAllPossibleCodeComparisonPanels(); - - // Put all panels in CodeComparisonPanel list; at same time, get a - // list of superseded panels. - ArrayList>> classesOfPanelsToSupersede = - new ArrayList<>(); - for (CodeComparisonPanel codeComparisonPanel : instances) { - codeComparisonPanels.add(codeComparisonPanel); - Class> panelThisSupersedes = - codeComparisonPanel.getPanelThisSupersedes(); - if (panelThisSupersedes != null) { - classesOfPanelsToSupersede.add(panelThisSupersedes); - } - } - - // Now go back through the panels and remove those that another one wants to supersede. - Iterator> iterator = - codeComparisonPanels.iterator(); - while (iterator.hasNext()) { - CodeComparisonPanel codeComparisonPanel = - iterator.next(); - if (classesOfPanelsToSupersede.contains(codeComparisonPanel.getClass())) { - // Remove the superseded panel. - iterator.remove(); - } - } - + codeComparisonPanels = createAllPossibleCodeComparisonPanels(); codeComparisonPanels.sort((p1, p2) -> p1.getTitle().compareTo(p2.getTitle())); } return codeComparisonPanels; } @SuppressWarnings({ "rawtypes", "unchecked" }) - private Set> createAllPossibleCodeComparisonPanels() { - Set> instances = new HashSet<>(); + private ArrayList> createAllPossibleCodeComparisonPanels() { + ArrayList> instances = + new ArrayList<>(); List> classes = ClassSearcher.getClasses(CodeComparisonPanel.class); for (Class panelClass : classes) { diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPlugin.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPlugin.java index 761d2b1b06..43d09d24cf 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPlugin.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonPlugin.java @@ -18,15 +18,19 @@ package ghidra.app.plugin.core.functioncompare; import java.util.Set; import java.util.function.Supplier; -import docking.ComponentProviderActivationListener; import ghidra.app.CorePluginPackage; -import ghidra.app.events.*; +import ghidra.app.events.ProgramActivatedPluginEvent; +import ghidra.app.events.ProgramClosedPluginEvent; +import ghidra.app.events.ProgramSelectionPluginEvent; import ghidra.app.plugin.PluginCategoryNames; import ghidra.app.plugin.ProgramPlugin; import ghidra.app.plugin.core.functioncompare.actions.CompareFunctionsAction; import ghidra.app.plugin.core.functioncompare.actions.CompareFunctionsFromListingAction; import ghidra.app.services.FunctionComparisonService; -import ghidra.framework.model.*; +import ghidra.framework.model.DomainObject; +import ghidra.framework.model.DomainObjectChangeRecord; +import ghidra.framework.model.DomainObjectChangedEvent; +import ghidra.framework.model.DomainObjectListener; import ghidra.framework.plugintool.PluginInfo; import ghidra.framework.plugintool.PluginTool; import ghidra.framework.plugintool.util.PluginStatus; @@ -57,7 +61,7 @@ import ghidra.util.Swing; ) //@formatter:on public class FunctionComparisonPlugin extends ProgramPlugin - implements DomainObjectListener, FunctionComparisonService { + implements DomainObjectListener, FunctionComparisonService { static final String MENU_PULLRIGHT = "CompareFunctions"; static final String POPUP_MENU_GROUP = "CompareFunction"; @@ -132,7 +136,7 @@ public class FunctionComparisonPlugin extends ProgramPlugin } private FunctionComparisonProvider getFromSwingBlocking( - Supplier comparer) { + Supplier comparer) { if (Swing.isSwingThread()) { return comparer.get(); @@ -141,22 +145,13 @@ public class FunctionComparisonPlugin extends ProgramPlugin return Swing.runNow(comparer); } + void providerClosed(FunctionComparisonProvider provider) { + functionComparisonManager.providerClosed(provider); + } //================================================================================================== // Service Methods //================================================================================================== - @Override - public void addFunctionComparisonProviderListener( - ComponentProviderActivationListener listener) { - runOnSwingNonBlocking(() -> functionComparisonManager.addProviderListener(listener)); - } - - @Override - public void removeFunctionComparisonProviderListener( - ComponentProviderActivationListener listener) { - runOnSwingNonBlocking(() -> functionComparisonManager.removeProviderListener(listener)); - } - @Override public void removeFunction(Function function) { runOnSwingNonBlocking(() -> functionComparisonManager.removeFunction(function)); @@ -174,7 +169,7 @@ public class FunctionComparisonPlugin extends ProgramPlugin @Override public FunctionComparisonProvider compareFunctions(Function source, - Function target) { + Function target) { return getFromSwingBlocking( () -> functionComparisonManager.compareFunctions(source, target)); } @@ -192,8 +187,9 @@ public class FunctionComparisonPlugin extends ProgramPlugin @Override public void compareFunctions(Function source, Function target, - FunctionComparisonProvider provider) { + FunctionComparisonProvider provider) { runOnSwingNonBlocking( () -> functionComparisonManager.compareFunctions(source, target, provider)); } + } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProvider.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProvider.java index 28ef35f134..b13bed4423 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProvider.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProvider.java @@ -31,7 +31,6 @@ import ghidra.app.util.viewer.listingpanel.ListingPanel; import ghidra.app.util.viewer.util.CodeComparisonPanel; import ghidra.framework.options.SaveState; import ghidra.framework.plugintool.ComponentProviderAdapter; -import ghidra.framework.plugintool.Plugin; import ghidra.program.model.listing.Function; import ghidra.program.model.listing.Program; import ghidra.util.HelpLocation; @@ -47,7 +46,7 @@ public class FunctionComparisonProvider extends ComponentProviderAdapter protected static final String HELP_TOPIC = "FunctionComparison"; protected FunctionComparisonPanel functionComparisonPanel; - protected Plugin plugin; + protected FunctionComparisonPlugin plugin; /** Contains all the comparison data to be displayed by this provider */ protected FunctionComparisonModel model; @@ -61,7 +60,7 @@ public class FunctionComparisonProvider extends ComponentProviderAdapter * the same window * @param owner the provider owner, usually a plugin name */ - public FunctionComparisonProvider(Plugin plugin, String name, String owner) { + public FunctionComparisonProvider(FunctionComparisonPlugin plugin, String name, String owner) { this(plugin, name, owner, null); } @@ -74,7 +73,7 @@ public class FunctionComparisonProvider extends ComponentProviderAdapter * @param owner the provider owner, usually a plugin name * @param contextType the type of context supported by this provider; may be null */ - public FunctionComparisonProvider(Plugin plugin, String name, String owner, + public FunctionComparisonProvider(FunctionComparisonPlugin plugin, String name, String owner, Class contextType) { super(plugin.getTool(), name, owner, contextType); this.plugin = plugin; @@ -97,6 +96,7 @@ public class FunctionComparisonProvider extends ComponentProviderAdapter public void closeComponent() { super.closeComponent(); closeListener.call(); + closeListener = Callback.dummy(); } @Override @@ -128,6 +128,7 @@ public class FunctionComparisonProvider extends ComponentProviderAdapter public void removeFromTool() { tool.removePopupActionProvider(this); super.removeFromTool(); + plugin.providerClosed(this); } @Override @@ -198,7 +199,7 @@ public class FunctionComparisonProvider extends ComponentProviderAdapter * @param functions the functions to remove */ public void removeFunctions(Set functions) { - functions.stream().forEach(f -> model.removeFunction(f)); + model.removeFunctions(functions); closeIfEmpty(); } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProviderManager.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProviderManager.java index 011149430c..143988e7e8 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProviderManager.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/FunctionComparisonProviderManager.java @@ -37,14 +37,14 @@ public class FunctionComparisonProviderManager implements FunctionComparisonProv private Set providers = new CopyOnWriteArraySet<>(); private Set listeners = new HashSet<>(); - private Plugin plugin; + private FunctionComparisonPlugin plugin; /** * Constructor * * @param plugin the parent plugin */ - public FunctionComparisonProviderManager(Plugin plugin) { + public FunctionComparisonProviderManager(FunctionComparisonPlugin plugin) { this.plugin = plugin; } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonPanel.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonPanel.java index b100e5b3dd..939d9dc7f3 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonPanel.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonPanel.java @@ -66,8 +66,7 @@ public class MultiFunctionComparisonPanel extends FunctionComparisonPanel { * @param provider the comparison provider associated with this panel * @param tool the active plugin tool */ - public MultiFunctionComparisonPanel(MultiFunctionComparisonProvider provider, - PluginTool tool) { + public MultiFunctionComparisonPanel(MultiFunctionComparisonProvider provider, PluginTool tool) { super(provider, tool, null, null); JPanel choicePanel = new JPanel(new GridLayout(1, 2)); @@ -184,6 +183,21 @@ public class MultiFunctionComparisonPanel extends FunctionComparisonPanel { } restoreSelection(targetFunctionsCB, selection); + + // we don't want the initial target to match the source as that is a pointless comparison + fixupTargetSelectionToNotMatchSource(source); + } + + private void fixupTargetSelectionToNotMatchSource(Function source) { + if (targetFunctionsCB.getSelectedItem() != source) { + return; + } + for (int i = 0; i < targetFunctionsCB.getItemCount(); i++) { + if (targetFunctionsCB.getItemAt(i) != source) { + targetFunctionsCB.setSelectedIndex(i); + return; + } + } } /** @@ -312,7 +326,7 @@ public class MultiFunctionComparisonPanel extends FunctionComparisonPanel { @Override public Component getListCellRendererComponent(JList list, Object value, int index, - boolean isSelected, boolean cellHasFocus) { + boolean isSelected, boolean cellHasFocus) { if (value == null) { // It's possible during a close program operation to have this @@ -329,9 +343,7 @@ public class MultiFunctionComparisonPanel extends FunctionComparisonPanel { String functionAddress = f.getBody().getMinAddress().toString(); String text = functionName + "@" + functionAddress + " (" + functionPathToProgram + ")"; - return super.getListCellRendererComponent(list, text, index, isSelected, - cellHasFocus); + return super.getListCellRendererComponent(list, text, index, isSelected, cellHasFocus); } } - } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonProvider.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonProvider.java index b65e860833..acf1c20f05 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonProvider.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/MultiFunctionComparisonProvider.java @@ -33,7 +33,7 @@ public class MultiFunctionComparisonProvider extends FunctionComparisonProvider * * @param plugin the parent plugin */ - protected MultiFunctionComparisonProvider(Plugin plugin) { + protected MultiFunctionComparisonProvider(FunctionComparisonPlugin plugin) { super(plugin, "Functions Comparison Provider", plugin.getName()); } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/AbstractApplyFunctionSignatureAction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/AbstractApplyFunctionSignatureAction.java deleted file mode 100644 index 09d7876003..0000000000 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/AbstractApplyFunctionSignatureAction.java +++ /dev/null @@ -1,158 +0,0 @@ -/* ### - * IP: GHIDRA - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package ghidra.app.plugin.core.functioncompare.actions; - -import docking.ActionContext; -import docking.ComponentProvider; -import docking.action.DockingAction; -import docking.action.MenuData; -import docking.widgets.fieldpanel.internal.FieldPanelCoordinator; -import ghidra.app.util.viewer.util.CodeComparisonPanel; -import ghidra.app.util.viewer.util.CodeComparisonPanelActionContext; -import ghidra.program.model.listing.*; -import ghidra.program.util.FunctionUtility; -import ghidra.util.*; -import ghidra.util.exception.DuplicateNameException; -import ghidra.util.exception.InvalidInputException; - -/** - * Applies the signature of the function in the currently active side of a - * code comparison panel to the function in the other side of the panel - *

- * Each CodeComparisonPanel can extend this class in order to provide this action - * using its context - */ -public abstract class AbstractApplyFunctionSignatureAction extends DockingAction { - - private static final String MENU_GROUP = "A0_Apply"; - private static final String HELP_TOPIC = "FunctionComparison"; - private static final String ACTION_NAME = "Apply Function Signature To Other Side"; - - /** - * Constructor - * - * @param owner the owner of this action - */ - public AbstractApplyFunctionSignatureAction(String owner) { - super(ACTION_NAME, owner); - - setDescription(HTMLUtilities.toHTML("Apply the signature of the function in the " + - "currently active side of a code comparison panel to the function in the other " + - "side of the panel.")); - MenuData menuData = new MenuData(new String[] { ACTION_NAME }, null, MENU_GROUP); - setPopupMenuData(menuData); - setEnabled(true); - setHelpLocation(new HelpLocation(HELP_TOPIC, ACTION_NAME)); - } - - @Override - public abstract boolean isAddToPopup(ActionContext context); - - @Override - public abstract boolean isEnabledForContext(ActionContext context); - - @Override - public void actionPerformed(ActionContext context) { - if (context instanceof CodeComparisonPanelActionContext) { - CodeComparisonPanelActionContext compareContext = - (CodeComparisonPanelActionContext) context; - CodeComparisonPanel codeComparisonPanel = - compareContext.getCodeComparisonPanel(); - Function leftFunction = codeComparisonPanel.getLeftFunction(); - Function rightFunction = codeComparisonPanel.getRightFunction(); - if (leftFunction == null || rightFunction == null) { - return; // Can only apply if both sides have functions. - } - ComponentProvider componentProvider = context.getComponentProvider(); - boolean leftHasFocus = codeComparisonPanel.leftPanelHasFocus(); - boolean commit; - if (leftHasFocus) { - commit = updateFunction(componentProvider, rightFunction, leftFunction); - } - else { - commit = updateFunction(componentProvider, leftFunction, rightFunction); - } - if (commit) { - // Refresh the side that had its function signature changed (the side without focus). - if (leftHasFocus) { - codeComparisonPanel.refreshRightPanel(); - } - else { - codeComparisonPanel.refreshLeftPanel(); - } - } - } - } - - /** - * Returns true if the comparison panel opposite the one with focus, - * is read-only - *

- * eg: if the right-side panel has focus, and the left-side panel is - * read-only, this will return true - * - * @param codeComparisonPanel the comparison panel - * @return true if the non-focused panel is read-only - */ - protected boolean hasReadOnlyNonFocusedSide( - CodeComparisonPanel codeComparisonPanel) { - Function leftFunction = codeComparisonPanel.getLeftFunction(); - Function rightFunction = codeComparisonPanel.getRightFunction(); - - if (leftFunction == null || rightFunction == null) { - return false; // Doesn't have a function on both sides. - } - - boolean leftHasFocus = codeComparisonPanel.leftPanelHasFocus(); - Program leftProgram = leftFunction.getProgram(); - Program rightProgram = rightFunction.getProgram(); - return (!leftHasFocus && leftProgram.getDomainFile().isReadOnly()) || - (leftHasFocus && rightProgram.getDomainFile().isReadOnly()); - } - - /** - * Attempts to change the signature of a function to that of another - * function - * - * @param provider the parent component provider - * @param destinationFunction the function to change - * @param sourceFunction the function to copy - * @return true if the operation was successful - */ - protected boolean updateFunction(ComponentProvider provider, Function destinationFunction, - Function sourceFunction) { - - Program program = destinationFunction.getProgram(); - int txID = program.startTransaction(ACTION_NAME); - boolean commit = false; - - try { - FunctionUtility.updateFunction(destinationFunction, sourceFunction); - commit = true; - } - catch (InvalidInputException | DuplicateNameException | CircularDependencyException e) { - String message = "Couldn't apply the function signature from " + - sourceFunction.getName() + " to " + destinationFunction.getName() + " @ " + - destinationFunction.getEntryPoint().toString() + ". " + e.getMessage(); - Msg.showError(this, provider.getComponent(), ACTION_NAME, message, e); - } - finally { - program.endTransaction(txID, commit); - } - - return commit; - } -} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/AbstractFunctionComparisonApplyAction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/AbstractFunctionComparisonApplyAction.java new file mode 100644 index 0000000000..de852b593c --- /dev/null +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/AbstractFunctionComparisonApplyAction.java @@ -0,0 +1,101 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.plugin.core.functioncompare.actions; + +import docking.ActionContext; +import docking.action.DockingAction; +import ghidra.app.util.viewer.util.CodeComparisonActionContext; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.util.Msg; + +/** + * Base classes for applying function information from a one side or the other in the function + * comparison window + */ +public abstract class AbstractFunctionComparisonApplyAction extends DockingAction { + protected static final String MENU_PARENT = "Apply From Other"; + protected static final String MENU_GROUP = "A0_Apply"; + protected static final String HELP_TOPIC = "FunctionComparison"; + + /** + * Constructor for base apply action + * @param name the name of the action + * @param owner the owner of the action + * the dual listing view or the dual decompiler view each of which produce their own action + * context types. Each different view creates their own version of each action using the + * context handler appropriate for that view. + */ + public AbstractFunctionComparisonApplyAction(String name, String owner) { + super(name, owner); + + } + + @Override + public boolean isValidContext(ActionContext context) { + return context instanceof CodeComparisonActionContext; + } + + @Override + public boolean isEnabledForContext(ActionContext context) { + if (context instanceof CodeComparisonActionContext comparisonContext) { + return isEnabledForContext(comparisonContext); + } + return false; + } + + private boolean isEnabledForContext(CodeComparisonActionContext context) { + Function source = context.getSourceFunction(); + Function target = context.getTargetFunction(); + if (source == null || target == null) { + return false; + } + return !target.getProgram().getDomainFile().isReadOnly(); + + } + + @Override + public void actionPerformed(ActionContext context) { + if (context instanceof CodeComparisonActionContext comparisonContext) { + doActionPerformed(comparisonContext); + } + } + + private void doActionPerformed(CodeComparisonActionContext context) { + Function source = context.getSourceFunction(); + Function target = context.getTargetFunction(); + Program program = target.getProgram(); + + try { + program.withTransaction(getName(), () -> applyFunctionData(source, target)); + } + catch (Exception e) { + String message = "Failed to apply " + source.getName() + " to " + target.getName() + + " @ " + target.getEntryPoint().toString() + ". " + e.getMessage(); + Msg.showError(this, null, getName(), message, e); + } + + } + + /** + * Subclasses override this method to apply function information from the source to the target. + * @param source the source function to get information from + * @param target the target function to apply information to + * @throws Exception throws a variety of exceptions depending on what is being applied and + * the apply fails. + */ + protected abstract void applyFunctionData(Function source, Function target) throws Exception; +} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/CompareFunctionsFromFunctionTableAction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/CompareFunctionsFromFunctionTableAction.java index 24b4bb7430..d9c86dd9df 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/CompareFunctionsFromFunctionTableAction.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/CompareFunctionsFromFunctionTableAction.java @@ -15,7 +15,10 @@ */ package ghidra.app.plugin.core.functioncompare.actions; -import java.util.*; +import java.util.Collections; +import java.util.HashSet; +import java.util.List; +import java.util.Set; import docking.ActionContext; import ghidra.app.plugin.core.functionwindow.FunctionRowObject; @@ -57,6 +60,13 @@ public class CompareFunctionsFromFunctionTableAction extends CompareFunctionsAct return isModelSupported(context); } + @Override + public boolean isEnabledForContext(ActionContext actionContext) { + GhidraTable table = (GhidraTable) actionContext.getContextObject(); + int[] selectedRows = table.getSelectedRows(); + return selectedRows.length > 1; + } + @Override protected Set getSelectedFunctions(ActionContext actionContext) { Set functions = new HashSet<>(); diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/EmptySignatureApplyAction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/EmptySignatureApplyAction.java new file mode 100644 index 0000000000..686b63bd69 --- /dev/null +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/EmptySignatureApplyAction.java @@ -0,0 +1,48 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.plugin.core.functioncompare.actions; + +import docking.action.MenuData; +import ghidra.program.model.listing.Function; +import ghidra.program.util.FunctionUtility; +import ghidra.util.HelpLocation; + +/** + * Action for applying skeleton function signatures from one function to another in the dual + * decompiler or dual listing view. By skeleton, we mean signatures where all complex data types + * (e.g., Structures) are replaced by empty placeholder data types. + */ +public class EmptySignatureApplyAction extends AbstractFunctionComparisonApplyAction { + + /** + * Constructor for applying skeleton function signatures action + * @param owner the action owner + */ + public EmptySignatureApplyAction(String owner) { + super("Function Comparison Apply Signature", owner); + MenuData menuData = + new MenuData(new String[] { MENU_PARENT, "Function Signature" }, null, MENU_GROUP); + menuData.setParentMenuGroup(MENU_GROUP); + setPopupMenuData(menuData); + setHelpLocation(new HelpLocation(HELP_TOPIC, getName())); + } + + @Override + protected void applyFunctionData(Function source, Function target) throws Exception { + FunctionUtility.applySignature(target, source, true, null); + } + +} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/FunctionNameApplyAction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/FunctionNameApplyAction.java new file mode 100644 index 0000000000..8c1c5d39c4 --- /dev/null +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/FunctionNameApplyAction.java @@ -0,0 +1,47 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.plugin.core.functioncompare.actions; + +import docking.action.MenuData; +import ghidra.program.model.listing.Function; +import ghidra.program.util.FunctionUtility; +import ghidra.util.HelpLocation; + +/** + * Action for applying function names and namespaces from one function to another in the dual + * decompiler or dual listing view. + */ +public class FunctionNameApplyAction extends AbstractFunctionComparisonApplyAction { + + /** + * Constructor for applying function name and namespace action + * @param owner the action owner + */ + public FunctionNameApplyAction(String owner) { + super("Function Comparison Apply Name", owner); + MenuData menuData = + new MenuData(new String[] { MENU_PARENT, "Function Name" }, null, MENU_GROUP); + menuData.setParentMenuGroup(MENU_GROUP); + setPopupMenuData(menuData); + setHelpLocation(new HelpLocation(HELP_TOPIC, getName())); + } + + @Override + protected void applyFunctionData(Function source, Function target) throws Exception { + FunctionUtility.applyNameAndNamespace(target, source); + } + +} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/SignatureWithDatatypesApplyAction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/SignatureWithDatatypesApplyAction.java new file mode 100644 index 0000000000..1f8e014df3 --- /dev/null +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functioncompare/actions/SignatureWithDatatypesApplyAction.java @@ -0,0 +1,47 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.plugin.core.functioncompare.actions; + +import docking.action.MenuData; +import ghidra.program.model.listing.Function; +import ghidra.program.util.FunctionUtility; +import ghidra.util.HelpLocation; + +/** + * Action for applying full function signatures and referenced data types from one function to + * another in the dual decompiler or dual listing view. + */ +public class SignatureWithDatatypesApplyAction extends AbstractFunctionComparisonApplyAction { + + /** + * Constructor for applying function signature and all its referenced data types action + * @param owner the action owner + */ + public SignatureWithDatatypesApplyAction(String owner) { + super("Function Comparison Apply Signature And Datatypes", owner); + MenuData menuData = new MenuData( + new String[] { MENU_PARENT, "Function Signature and Data Types" }, null, MENU_GROUP); + menuData.setParentMenuGroup(MENU_GROUP); + setPopupMenuData(menuData); + setHelpLocation(new HelpLocation(HELP_TOPIC, getName())); + } + + @Override + protected void applyFunctionData(Function source, Function target) throws Exception { + FunctionUtility.applySignature(target, source, false, null); + } + +} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functionwindow/FunctionWindowPlugin.java b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functionwindow/FunctionWindowPlugin.java index 07a9aaeafb..134c2b6d7c 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functionwindow/FunctionWindowPlugin.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/plugin/core/functionwindow/FunctionWindowPlugin.java @@ -15,14 +15,11 @@ */ package ghidra.app.plugin.core.functionwindow; -import docking.ComponentProvider; -import docking.ComponentProviderActivationListener; import docking.action.DockingAction; import ghidra.app.CorePluginPackage; import ghidra.app.events.ProgramClosedPluginEvent; import ghidra.app.plugin.PluginCategoryNames; import ghidra.app.plugin.ProgramPlugin; -import ghidra.app.plugin.core.functioncompare.FunctionComparisonProvider; import ghidra.app.plugin.core.functioncompare.actions.CompareFunctionsFromFunctionTableAction; import ghidra.app.services.FunctionComparisonService; import ghidra.framework.model.*; @@ -49,14 +46,12 @@ import ghidra.util.task.SwingUpdateManager; eventsConsumed = { ProgramClosedPluginEvent.class } ) //@formatter:on -public class FunctionWindowPlugin extends ProgramPlugin implements DomainObjectListener, - ComponentProviderActivationListener { +public class FunctionWindowPlugin extends ProgramPlugin implements DomainObjectListener { private DockingAction selectAction; private DockingAction compareFunctionsAction; private FunctionWindowProvider provider; private SwingUpdateManager swingMgr; - private FunctionComparisonService functionComparisonService; public FunctionWindowPlugin(PluginTool tool) { super(tool); @@ -94,20 +89,17 @@ public class FunctionWindowPlugin extends ProgramPlugin implements DomainObjectL @Override public void serviceAdded(Class interfaceClass, Object service) { if (interfaceClass == FunctionComparisonService.class) { - functionComparisonService = (FunctionComparisonService) service; - - // Listen for providers being opened/closed to we can disable - // comparison actions if there are no comparison providers - // open - functionComparisonService.addFunctionComparisonProviderListener(this); + compareFunctionsAction = new CompareFunctionsFromFunctionTableAction(tool, getName()); + tool.addLocalAction(provider, compareFunctionsAction); + tool.contextChanged(provider); } } @Override public void serviceRemoved(Class interfaceClass, Object service) { if (interfaceClass == FunctionComparisonService.class) { - functionComparisonService.removeFunctionComparisonProviderListener(this); - functionComparisonService = null; + tool.removeLocalAction(provider, compareFunctionsAction); + compareFunctionsAction = null; } } @@ -207,25 +199,12 @@ public class FunctionWindowPlugin extends ProgramPlugin implements DomainObjectL selectAction = new MakeProgramSelectionAction(this, provider.getTable()); tool.addLocalAction(provider, selectAction); - compareFunctionsAction = new CompareFunctionsFromFunctionTableAction(tool, getName()); - tool.addLocalAction(provider, compareFunctionsAction); + // note that the compare functions action is only added when the compare functions service + // is added to the tool } void showFunctions() { provider.showFunctions(); } - @Override - public void componentProviderActivated(ComponentProvider componentProvider) { - if (componentProvider instanceof FunctionComparisonProvider) { - tool.contextChanged(provider); - } - } - - @Override - public void componentProviderDeactivated(ComponentProvider componentProvider) { - if (componentProvider instanceof FunctionComparisonProvider) { - tool.contextChanged(provider); - } - } } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonModel.java b/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonModel.java index 9ab14fd7b1..6490ef423f 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonModel.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonModel.java @@ -150,6 +150,22 @@ public class FunctionComparisonModel { * @param function the function to remove */ public void removeFunction(Function function) { + doRemoveFunction(function); + fireModelChanged(); + } + + /** + * Removes all the given functions from all comparisons in the model + * @param functions the functions to remove + */ + public void removeFunctions(Collection functions) { + for (Function function : functions) { + doRemoveFunction(function); + } + fireModelChanged(); + } + + private void doRemoveFunction(Function function) { List comparisonsToRemove = new ArrayList<>(); Iterator iter = comparisons.iterator(); @@ -167,8 +183,6 @@ public class FunctionComparisonModel { } comparisons.removeAll(comparisonsToRemove); - - fireModelChanged(); } /** @@ -178,19 +192,14 @@ public class FunctionComparisonModel { * @param program the program to remove functions from */ public void removeFunctions(Program program) { - Set sources = getSourceFunctions(); - Set targets = getTargetFunctions(); + Set allFunctions = getSourceFunctions(); + allFunctions.addAll(getTargetFunctions()); - Set sourcesToRemove = sources.stream() + Set functionsToRemove = allFunctions.stream() .filter(f -> f.getProgram().equals(program)) .collect(Collectors.toSet()); - Set targetsToRemove = targets.stream() - .filter(f -> f.getProgram().equals(program)) - .collect(Collectors.toSet()); - - sourcesToRemove.stream().forEach(f -> removeFunction(f)); - targetsToRemove.stream().forEach(f -> removeFunction(f)); + removeFunctions(functionsToRemove); } /** @@ -280,8 +289,7 @@ public class FunctionComparisonModel { // Remove any functions that already have an comparison in the // model; these will be ignored - functions.removeIf(f -> comparisons.stream() - .anyMatch(fc -> f.equals(fc.getSource()))); + functions.removeIf(f -> comparisons.stream().anyMatch(fc -> f.equals(fc.getSource()))); monitor.setIndeterminate(false); monitor.setMessage("Creating new comparisons"); diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonService.java b/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonService.java index d3059f5199..833b1550c7 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonService.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/services/FunctionComparisonService.java @@ -17,7 +17,6 @@ package ghidra.app.services; import java.util.Set; -import docking.ComponentProviderActivationListener; import ghidra.app.plugin.core.functioncompare.FunctionComparisonPlugin; import ghidra.app.plugin.core.functioncompare.FunctionComparisonProvider; import ghidra.framework.plugintool.ServiceInfo; @@ -91,7 +90,7 @@ public interface FunctionComparisonService { * @param provider the provider to add the comparisons to */ public void compareFunctions(Set functions, - FunctionComparisonProvider provider); + FunctionComparisonProvider provider); /** * Creates a comparison between two functions and adds it to a given @@ -106,7 +105,7 @@ public interface FunctionComparisonService { * @param provider the provider to add the comparison to */ public void compareFunctions(Function source, Function target, - FunctionComparisonProvider provider); + FunctionComparisonProvider provider); /** * Removes a given function from all comparisons across all comparison @@ -124,20 +123,4 @@ public interface FunctionComparisonService { * @param provider the comparison provider to remove functions from */ public void removeFunction(Function function, FunctionComparisonProvider provider); - - /** - * Adds the given listener to the list of subscribers who wish to be - * notified of provider activation events (eg: provider open/close) - * - * @param listener the listener to be added - */ - public void addFunctionComparisonProviderListener(ComponentProviderActivationListener listener); - - /** - * Removes a listener from the list of provider activation event subscribers - * - * @param listener the listener to remove - */ - public void removeFunctionComparisonProviderListener( - ComponentProviderActivationListener listener); } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/demangler/DemangledFunction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/demangler/DemangledFunction.java index ca0838979f..a514ffb900 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/demangler/DemangledFunction.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/demangler/DemangledFunction.java @@ -454,7 +454,8 @@ public class DemangledFunction extends DemangledObject { } ApplyFunctionSignatureCmd cmd = new ApplyFunctionSignatureCmd(function.getEntryPoint(), - signature, SourceType.IMPORTED, true, FunctionRenameOption.RENAME_IF_DEFAULT); + signature, SourceType.IMPORTED, true, false, DataTypeConflictHandler.DEFAULT_HANDLER, + FunctionRenameOption.RENAME_IF_DEFAULT); cmd.applyTo(program); return true; @@ -594,8 +595,8 @@ public class DemangledFunction extends DemangledObject { return null; } - private Structure maybeUpdateCallingConventionAndCreateClass(Program program, - Function function, DemanglerOptions options) { + private Structure maybeUpdateCallingConventionAndCreateClass(Program program, Function function, + DemanglerOptions options) { String convention = validateCallingConvention(program, function, options); if (convention == null) { diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ApplyFunctionSignatureAction.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ApplyFunctionSignatureAction.java deleted file mode 100644 index b8fb6d529f..0000000000 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ApplyFunctionSignatureAction.java +++ /dev/null @@ -1,55 +0,0 @@ -/* ### - * IP: GHIDRA - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package ghidra.app.util.viewer.listingpanel; - -import docking.ActionContext; -import docking.widgets.fieldpanel.internal.FieldPanelCoordinator; -import ghidra.app.plugin.core.functioncompare.actions.AbstractApplyFunctionSignatureAction; -import ghidra.app.util.viewer.util.CodeComparisonPanel; - -/** - * Action that applies the signature of the function in the currently active side of a listing - * code comparison panel to the function in the other side of the panel. - */ -public class ApplyFunctionSignatureAction extends AbstractApplyFunctionSignatureAction { - - /** - * Constructor for the action that applies a function signature from one side of a dual - * listing panel to the other. - * @param owner the owner of this action. - */ - public ApplyFunctionSignatureAction(String owner) { - super(owner); - } - - @Override - public boolean isAddToPopup(ActionContext context) { - return (context instanceof DualListingActionContext); - } - - @Override - public boolean isEnabledForContext(ActionContext context) { - if (context instanceof DualListingActionContext) { - DualListingActionContext compareContext = (DualListingActionContext) context; - CodeComparisonPanel codeComparisonPanel = - compareContext.getCodeComparisonPanel(); - if (codeComparisonPanel instanceof ListingCodeComparisonPanel) { - return !hasReadOnlyNonFocusedSide(codeComparisonPanel); - } - } - return false; - } -} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/DualListingActionContext.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/DualListingActionContext.java index 2fe113ffa2..b1e61b956f 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/DualListingActionContext.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/DualListingActionContext.java @@ -17,13 +17,9 @@ package ghidra.app.util.viewer.listingpanel; import docking.ComponentProvider; import docking.widgets.fieldpanel.internal.FieldPanelCoordinator; -import ghidra.app.context.NavigatableActionContext; -import ghidra.app.nav.Navigatable; +import ghidra.app.util.viewer.util.CodeComparisonActionContext; import ghidra.app.util.viewer.util.CodeComparisonPanel; -import ghidra.app.util.viewer.util.CodeComparisonPanelActionContext; -import ghidra.program.model.listing.Program; -import ghidra.program.util.ProgramLocation; -import ghidra.program.util.ProgramSelection; +import ghidra.program.model.listing.Function; // Note: If you want to get the typical actions for things like comments, labels, bookmarks, etc. // that are available in the CodeBrowser then change this to extend ListingActionContext. @@ -32,44 +28,16 @@ import ghidra.program.util.ProgramSelection; /** * Action context for a ListingCodeComparisonPanel. */ -public class DualListingActionContext extends NavigatableActionContext - implements CodeComparisonPanelActionContext { +public class DualListingActionContext extends CodeComparisonActionContext { private CodeComparisonPanel codeComparisonPanel = null; /** * Constructor for a dual listing's action context. * @param provider the provider that uses this action context. - * @param navigatable the navigatable for this action context. - * @param program the program in the listing providing this context. - * @param location the location indicated by this context. - * @param selection the listing selection for this context. - * @param highlight the listing highlight for this context. */ - public DualListingActionContext(ComponentProvider provider, Navigatable navigatable, - Program program, ProgramLocation location, ProgramSelection selection, - ProgramSelection highlight) { - super(provider, navigatable, program, location, selection, highlight); - } - - /** - * Constructor for a dual listing's action context. - * @param provider the provider that uses this action context. - * @param navigatable the navigatable for this action context. - * @param location the location indicated by this context. - */ - public DualListingActionContext(ComponentProvider provider, Navigatable navigatable, - ProgramLocation location) { - super(provider, navigatable, location); - } - - /** - * Constructor for a dual listing's action context. - * @param provider the provider that uses this action context. - * @param navigatable the navigatable for this action context. - */ - public DualListingActionContext(ComponentProvider provider, Navigatable navigatable) { - super(provider, navigatable); + public DualListingActionContext(ComponentProvider provider) { + super(provider); } /** @@ -85,4 +53,20 @@ public class DualListingActionContext extends NavigatableActionContext public CodeComparisonPanel getCodeComparisonPanel() { return codeComparisonPanel; } + + @Override + public Function getSourceFunction() { + boolean leftHasFocus = codeComparisonPanel.leftPanelHasFocus(); + + return leftHasFocus ? codeComparisonPanel.getRightFunction() + : codeComparisonPanel.getLeftFunction(); + } + + @Override + public Function getTargetFunction() { + boolean leftHasFocus = codeComparisonPanel.leftPanelHasFocus(); + + return leftHasFocus ? codeComparisonPanel.getLeftFunction() + : codeComparisonPanel.getRightFunction(); + } } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonOptions.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonOptions.java index ad0aa22cad..dce08520e7 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonOptions.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonOptions.java @@ -24,7 +24,7 @@ import ghidra.util.HelpLocation; public class ListingCodeComparisonOptions { public static final String OPTIONS_CATEGORY_NAME = "Listing Code Comparison"; - public static final String HELP_TOPIC = "Listing Code Comparison"; + public static final String HELP_TOPIC = "FunctionComparison"; private static final String BYTE_DIFFS_COLOR_KEY = "Byte Differences Color"; private static final String MNEMONIC_DIFFS_COLOR_KEY = "Mnemonic Differences Color"; @@ -101,7 +101,7 @@ public class ListingCodeComparisonOptions { } public void initializeOptions(ToolOptions options) { - HelpLocation help = new HelpLocation(HELP_TOPIC, "Options"); + HelpLocation help = new HelpLocation(HELP_TOPIC, "Listing Code Comparison Options"); options.setOptionsHelpLocation(help); options.registerThemeColorBinding(BYTE_DIFFS_COLOR_KEY, "color.bg.listing.comparison.bytes", diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonPanel.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonPanel.java index c019d893f1..54beae9885 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonPanel.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/listingpanel/ListingCodeComparisonPanel.java @@ -42,6 +42,7 @@ import ghidra.GhidraOptions; import ghidra.app.nav.Navigatable; import ghidra.app.plugin.core.codebrowser.MarkerServiceBackgroundColorModel; import ghidra.app.plugin.core.codebrowser.hover.*; +import ghidra.app.plugin.core.functioncompare.actions.*; import ghidra.app.plugin.core.marker.MarkerManager; import ghidra.app.services.*; import ghidra.app.util.ListingHighlightProvider; @@ -134,7 +135,9 @@ public class ListingCodeComparisonPanel private PreviousDiffAction previousDiffAction; private ListingCodeComparisonOptionsAction optionsAction; private DockingAction[] diffActions; - private ApplyFunctionSignatureAction applyFunctionSignatureAction; + private FunctionNameApplyAction applyNameAction; + private EmptySignatureApplyAction applySignatureEmptyDatatypesAction; + private SignatureWithDatatypesApplyAction applySignatureWithDatatypesAction; private JSplitPane splitPane; private ListingDiffHighlightProvider leftDiffHighlightProvider; @@ -268,7 +271,7 @@ public class ListingCodeComparisonPanel for (int i = 0; i < 2; i++) { fieldPanels[i] = listingPanels[i].getFieldPanel(); fieldPanels[i].addFocusListener(this); - fieldPanels[i].addMouseListener(new DualListingMouseListener(fieldPanels[i], i)); + fieldPanels[i].addMouseListener(new DualListingMouseListener(i)); } leftLocationListener = new LeftLocationListener(); @@ -448,7 +451,9 @@ public class ListingCodeComparisonPanel toggleHeaderAction = new ToggleHeaderAction(); toggleOrientationAction = new ToggleOrientationAction(); toggleHoverAction = new ToggleHoverAction(); - applyFunctionSignatureAction = new ApplyFunctionSignatureAction(owner); + applyNameAction = new FunctionNameApplyAction(owner); + applySignatureEmptyDatatypesAction = new EmptySignatureApplyAction(owner); + applySignatureWithDatatypesAction = new SignatureWithDatatypesApplyAction(owner); nextDiffAction = new NextDiffAction(); previousDiffAction = new PreviousDiffAction(); optionsAction = new ListingCodeComparisonOptionsAction(); @@ -465,7 +470,8 @@ public class ListingCodeComparisonPanel public DockingAction[] getActions() { DockingAction[] codeCompActions = super.getActions(); DockingAction[] otherActions = new DockingAction[] { toggleHeaderAction, - toggleOrientationAction, toggleHoverAction, applyFunctionSignatureAction, + toggleOrientationAction, toggleHoverAction, applyNameAction, + applySignatureEmptyDatatypesAction, applySignatureWithDatatypesAction, nextPreviousAreaMarkerAction, nextDiffAction, previousDiffAction, optionsAction }; int compCount = codeCompActions.length; int otherCount = otherActions.length; @@ -1540,7 +1546,7 @@ public class ListingCodeComparisonPanel MarginProvider marginProvider = markerManagers[leftOrRight].getMarginProvider(); JComponent providerComp = marginProvider.getComponent(); DualListingMouseListener providerMouseListener = - new DualListingMouseListener(providerComp, leftOrRight); + new DualListingMouseListener(leftOrRight); providerComp.addMouseListener(providerMouseListener); listingPanels[leftOrRight].addMarginProvider(marginProvider); @@ -1548,7 +1554,7 @@ public class ListingCodeComparisonPanel OverviewProvider overviewProvider = markerManagers[leftOrRight].getOverviewProvider(); JComponent overviewComp = overviewProvider.getComponent(); DualListingMouseListener overviewMouseListener = - new DualListingMouseListener(overviewComp, leftOrRight); + new DualListingMouseListener(leftOrRight); overviewComp.addMouseListener(overviewMouseListener); listingPanels[leftOrRight].addOverviewProvider(overviewProvider); } @@ -1561,8 +1567,8 @@ public class ListingCodeComparisonPanel if (programs[LEFT] != null) { AddressIndexMap indexMap = listingPanels[LEFT].getAddressIndexMap(); listingPanels[LEFT].getFieldPanel() - .setBackgroundColorModel(new MarkerServiceBackgroundColorModel( - markerManagers[LEFT], programs[LEFT], indexMap)); + .setBackgroundColorModel(new MarkerServiceBackgroundColorModel( + markerManagers[LEFT], programs[LEFT], indexMap)); unmatchedCodeMarkers[LEFT] = markerManagers[LEFT].createAreaMarker("Listing1 Unmatched Code", "Instructions that are not matched to an instruction in the other function.", @@ -1575,8 +1581,8 @@ public class ListingCodeComparisonPanel if (programs[RIGHT] != null) { AddressIndexMap rightIndexMap = listingPanels[RIGHT].getAddressIndexMap(); listingPanels[RIGHT].getFieldPanel() - .setBackgroundColorModel(new MarkerServiceBackgroundColorModel( - markerManagers[RIGHT], programs[RIGHT], rightIndexMap)); + .setBackgroundColorModel(new MarkerServiceBackgroundColorModel( + markerManagers[RIGHT], programs[RIGHT], rightIndexMap)); unmatchedCodeMarkers[RIGHT] = markerManagers[RIGHT].createAreaMarker("Listing2 Unmatched Code", "Instructions that are not matched to an instruction in the other function.", @@ -1676,8 +1682,8 @@ public class ListingCodeComparisonPanel indexMaps[LEFT] = new AddressIndexMap(addressSets[LEFT]); markerManagers[LEFT].getOverviewProvider().setProgram(getLeftProgram(), indexMaps[LEFT]); listingPanels[LEFT].getFieldPanel() - .setBackgroundColorModel(new MarkerServiceBackgroundColorModel(markerManagers[LEFT], - programs[LEFT], indexMaps[LEFT])); + .setBackgroundColorModel(new MarkerServiceBackgroundColorModel(markerManagers[LEFT], + programs[LEFT], indexMaps[LEFT])); } private void updateRightAddressSet(Function rightFunction) { @@ -1693,8 +1699,8 @@ public class ListingCodeComparisonPanel indexMaps[RIGHT] = new AddressIndexMap(addressSets[RIGHT]); markerManagers[RIGHT].getOverviewProvider().setProgram(getRightProgram(), indexMaps[RIGHT]); listingPanels[RIGHT].getFieldPanel() - .setBackgroundColorModel(new MarkerServiceBackgroundColorModel( - markerManagers[RIGHT], programs[RIGHT], indexMaps[RIGHT])); + .setBackgroundColorModel(new MarkerServiceBackgroundColorModel( + markerManagers[RIGHT], programs[RIGHT], indexMaps[RIGHT])); } @Override @@ -2072,9 +2078,7 @@ public class ListingCodeComparisonPanel ListingCodeComparisonPanel dualListingPanel = this; if (event == null) { - Navigatable focusedNavigatable = dualListingPanel.getFocusedNavigatable(); - DualListingActionContext myActionContext = - new DualListingActionContext(provider, focusedNavigatable); + DualListingActionContext myActionContext = new DualListingActionContext(provider); myActionContext.setContextObject(this); myActionContext.setCodeComparisonPanel(this); return myActionContext; @@ -2099,9 +2103,7 @@ public class ListingCodeComparisonPanel return new DefaultActionContext(provider).setContextObject(fieldHeaderLocation); } - Navigatable focusedNavigatable = dualListingPanel.getFocusedNavigatable(); - DualListingActionContext myActionContext = - new DualListingActionContext(provider, focusedNavigatable); + DualListingActionContext myActionContext = new DualListingActionContext(provider); myActionContext.setContextObject(this); myActionContext.setCodeComparisonPanel(this); myActionContext.setSourceObject(source); @@ -2579,11 +2581,6 @@ public class ListingCodeComparisonPanel return data[RIGHT]; } - @Override - public Class> getPanelThisSupersedes() { - return null; // Doesn't supersede any other panel. - } - private class DualListingMarkerManager extends MarkerManager { private DualListingServiceProvider serviceProvider; @@ -2653,16 +2650,12 @@ public class ListingCodeComparisonPanel private int leftOrRight; - @SuppressWarnings("unused") - private Component leftOrRightComponent; - - DualListingMouseListener(Component leftOrRightComponent, int leftOrRight) { - this.leftOrRightComponent = leftOrRightComponent; + DualListingMouseListener(int leftOrRight) { this.leftOrRight = leftOrRight; } @Override - public void mouseClicked(MouseEvent e) { + public void mousePressed(MouseEvent e) { setDualPanelFocus(leftOrRight); } } @@ -2690,14 +2683,14 @@ public class ListingCodeComparisonPanel Object sourceMarginContextObject = getContextObjectForMarginPanels(sourcePanel, event); if (sourceMarginContextObject != null) { return new DefaultActionContext(provider) - .setContextObject(sourceMarginContextObject); + .setContextObject(sourceMarginContextObject); } // Are we on a marker margin of the right listing? Return that margin's context. Object destinationMarginContextObject = getContextObjectForMarginPanels(destinationPanel, event); if (destinationMarginContextObject != null) { return new DefaultActionContext(provider) - .setContextObject(destinationMarginContextObject); + .setContextObject(destinationMarginContextObject); } // If the action is on the Field Header of the left listing panel return an diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/util/CodeComparisonActionContext.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/util/CodeComparisonActionContext.java new file mode 100644 index 0000000000..acf9089ca7 --- /dev/null +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/util/CodeComparisonActionContext.java @@ -0,0 +1,62 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.util.viewer.util; + +import java.awt.Component; + +import docking.ComponentProvider; +import docking.DefaultActionContext; +import ghidra.program.model.listing.Function; + +public abstract class CodeComparisonActionContext extends DefaultActionContext + implements CodeComparisonPanelActionContext { + /** + * Constructor with no source component and no context object + * @param provider the ComponentProvider that generated this context. + */ + public CodeComparisonActionContext(ComponentProvider provider) { + super(provider); + } + + /** + * Constructor with source component and context object + * + * @param provider the ComponentProvider that generated this context. + * @param contextObject an optional contextObject that the ComponentProvider can provide; this + * can be anything that actions wish to later retrieve + * @param sourceComponent an optional source object; this is intended to be the component that + * is the source of the context, usually the focused component + */ + public CodeComparisonActionContext(ComponentProvider provider, Object contextObject, + Component sourceComponent) { + super(provider, contextObject, sourceComponent); + } + + /** + * Returns the function that is the source of the info being applied. This will be whichever + * side of the function diff window that isn't active. + * @return the function to get information from + */ + public abstract Function getSourceFunction(); + + /** + * Returns the function that is the target of the info being applied. This will be whichever + * side of the function diff window that is active. + * @return the function to apply information to + */ + public abstract Function getTargetFunction(); + +} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/util/CodeComparisonPanel.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/util/CodeComparisonPanel.java index 17bbc5d824..75da8e4282 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/util/CodeComparisonPanel.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/viewer/util/CodeComparisonPanel.java @@ -172,14 +172,6 @@ public abstract class CodeComparisonPanel exten this.showTitles = showTitles; } - /** - * Determines if this panel is intended to take the place of another and if so it returns - * the class of the panel to be superseded. - * @return the class for the CodeComparisonPanel that this one supersedes - * or null if it doesn't supersede another panel. - */ - public abstract Class> getPanelThisSupersedes(); - /** * Returns the context object which corresponds to the area of focus within this provider's * component. Null is returned when there is no context. @@ -354,7 +346,6 @@ public abstract class CodeComparisonPanel exten } this.syncScrolling = syncScrolling; - // Refresh the left panel. FieldPanel leftPanel = getLeftFieldPanel(); leftPanel.validate(); @@ -364,7 +355,6 @@ public abstract class CodeComparisonPanel exten rightPanel.validate(); rightPanel.invalidate(); - setFieldPanelCoordinator(syncScrolling ? createFieldPanelCoordinator() : null); } } diff --git a/Ghidra/Features/Base/src/main/java/ghidra/app/util/xml/FunctionsXmlMgr.java b/Ghidra/Features/Base/src/main/java/ghidra/app/util/xml/FunctionsXmlMgr.java index 2f6cd561ce..d64ec1df3d 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/app/util/xml/FunctionsXmlMgr.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/app/util/xml/FunctionsXmlMgr.java @@ -121,15 +121,14 @@ class FunctionsXmlMgr { AddressSet body = new AddressSet(entryPoint, entryPoint); if (functionElement.hasAttribute("LIBRARY_FUNCTION")) { - boolean isLibFunc = XmlUtilities.parseBoolean( - functionElement.getAttribute("LIBRARY_FUNCTION")); + boolean isLibFunc = XmlUtilities + .parseBoolean(functionElement.getAttribute("LIBRARY_FUNCTION")); if (isLibFunc) { BookmarkManager bm = program.getBookmarkManager(); BookmarkType bt = bm.getBookmarkType("IMPORTED"); if (bt == null) { Icon icon = new GIcon("icon.base.util.xml.functions.bookmark"); - bt = bm.defineType("IMPORTED", icon, Palette.DARK_GRAY, - 0); + bt = bm.defineType("IMPORTED", icon, Palette.DARK_GRAY, 0); } bm.setBookmark(entryPoint, "IMPORTED", LIB_BOOKMARK_CATEGORY, "Library function"); @@ -152,9 +151,8 @@ class FunctionsXmlMgr { program.getAddressFactory())) { try { Symbol symbol = func.getSymbol(); - Namespace namespace = - NamespaceUtils.getFunctionNamespaceAt(program, namespacePath, - entryPoint); + Namespace namespace = NamespaceUtils.getFunctionNamespaceAt(program, + namespacePath, entryPoint); if (namespace == null) { namespace = program.getGlobalNamespace(); } @@ -237,15 +235,16 @@ class FunctionsXmlMgr { private void tryToParseTypeInfoComment(TaskMonitor monitor, Function func, String typeInfoComment) { try { - FunctionDefinitionDataType funcDef = CParserUtils.parseSignature( - (DataTypeManagerService) null, program, typeInfoComment, false); + FunctionDefinitionDataType funcDef = CParserUtils + .parseSignature((DataTypeManagerService) null, program, typeInfoComment, false); if (funcDef == null) { log.appendMsg("Unable to parse function definition: " + typeInfoComment); return; } ApplyFunctionSignatureCmd afsCmd = new ApplyFunctionSignatureCmd(func.getEntryPoint(), - funcDef, SourceType.IMPORTED, false, FunctionRenameOption.RENAME_IF_DEFAULT); + funcDef, SourceType.IMPORTED, false, false, DataTypeConflictHandler.DEFAULT_HANDLER, + FunctionRenameOption.RENAME_IF_DEFAULT); if (!afsCmd.applyTo(program, monitor)) { // TODO: continue trying to add local vars after failing to update the function signature? log.appendMsg("Failed to update function " + funcDesc(func) + " with signature \"" + @@ -269,8 +268,7 @@ class FunctionsXmlMgr { String name = v.getName(); boolean isDefaultVariableName = (name == null) || SymbolUtilities.getDefaultLocalName(program, v.getStackOffset(), 0) - .equals( - name); + .equals(name); SourceType sourceType = isDefaultVariableName ? SourceType.DEFAULT : SourceType.USER_DEFINED; diff --git a/Ghidra/Features/Base/src/main/java/ghidra/program/util/DataTypeCleaner.java b/Ghidra/Features/Base/src/main/java/ghidra/program/util/DataTypeCleaner.java new file mode 100644 index 0000000000..53b0a82905 --- /dev/null +++ b/Ghidra/Features/Base/src/main/java/ghidra/program/util/DataTypeCleaner.java @@ -0,0 +1,137 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.program.util; + +import java.io.Closeable; +import java.util.Iterator; + +import ghidra.program.database.data.DataTypeUtilities; +import ghidra.program.model.data.*; +import ghidra.program.model.data.StandAloneDataTypeManager.LanguageUpdateOption; +import ghidra.program.model.lang.ProgramArchitecture; +import ghidra.util.task.TaskMonitor; + +/** + * {@link DataTypeCleaner} provides a convenient way to clean composite definitions which may be + * included within a complex datatype which was derived from an source unrelated to a target + * {@link DataTypeManager}. The cleaning process entails clearing all details associated with + * all composites other than their description which may be present. There is also an option + * to retain those composites which are already defined within the target. + *
+ * All datatypes and their referenced datatypes will be accumulated and possibly re-used across + * multiple invocations of the {@link #clean(DataType)} method. It is important that this instance + * be {@link #close() closed} when instance and any resulting {@link DataType} is no longer in use. + */ +public class DataTypeCleaner implements Closeable { + + private final DataTypeManager targetDtm; + private final boolean retainExistingComposites; + private final StandAloneDataTypeManager cleanerDtm; + + private int txId; + + /** + * Consruct a {@link DataTypeCleaner} instance. The caller must ensure that this instance + * is {@link #close() closed} when instance and any resulting {@link DataType} is no longer in + * use. + * @param targetDtm target datatype manager + * @param retainExistingComposites if true all composites will be checked against the + * {@code targetDtm} and retained if it already exists, otherwise all composites will be + * cleaned. + */ + public DataTypeCleaner(DataTypeManager targetDtm, boolean retainExistingComposites) { + this.targetDtm = targetDtm; + this.retainExistingComposites = retainExistingComposites; + this.cleanerDtm = new StandAloneDataTypeManager("CleanerDTM"); + txId = cleanerDtm.startTransaction("CleanerTx"); + + ProgramArchitecture arch = targetDtm.getProgramArchitecture(); + if (arch != null) { + try { + cleanerDtm.setProgramArchitecture(arch.getLanguage(), + arch.getCompilerSpec().getCompilerSpecID(), LanguageUpdateOption.UNCHANGED, + TaskMonitor.DUMMY); + } + catch (Exception e) { + throw new RuntimeException(e); + } + } + } + + /** + * Clean the specified datatype + * @param dt datatype + * @return clean datatype + */ + public DataType clean(DataType dt) { + + if (txId == -1) { + throw new IllegalStateException("DataTypeCleaner has been closed"); + } + + DataType cleanDt = cleanerDtm.resolve(dt, + DataTypeConflictHandler.REPLACE_EMPTY_STRUCTS_OR_RENAME_AND_ADD_HANDLER); + + Iterator allComposites = cleanerDtm.getAllComposites(); + while (allComposites.hasNext()) { + Composite c = allComposites.next(); + if (c.isNotYetDefined()) { + continue; + } + if (retainExistingComposites && targetContainsComposite(c)) { + continue; + } + Composite replacement = null; + if (c instanceof Structure s) { + replacement = + new StructureDataType(c.getCategoryPath(), c.getName(), 0, cleanerDtm); + } + else if (c instanceof Union u) { + replacement = new UnionDataType(c.getCategoryPath(), c.getName(), cleanerDtm); + } + if (replacement != null) { + replacement.setDescription(c.getDescription()); + c.replaceWith(replacement); + } + } + return cleanDt; + } + + private boolean targetContainsComposite(Composite c) { + Category category = targetDtm.getCategory(c.getCategoryPath()); + if (category == null) { + return false; + } + String baseName = DataTypeUtilities.getNameWithoutConflict(c, false); + for (DataType dt : category.getDataTypesByBaseName(baseName)) { + if (dt.isEquivalent(c)) { + return true; + } + } + return false; + } + + @Override + public void close() { + if (txId == -1) { + return; + } + cleanerDtm.endTransaction(txId, true); // faster to commit + cleanerDtm.close(); + txId = -1; // closed indicator + } + +} diff --git a/Ghidra/Features/Base/src/main/java/ghidra/program/util/FunctionUtility.java b/Ghidra/Features/Base/src/main/java/ghidra/program/util/FunctionUtility.java index 95ac91a1c6..7f5d780013 100644 --- a/Ghidra/Features/Base/src/main/java/ghidra/program/util/FunctionUtility.java +++ b/Ghidra/Features/Base/src/main/java/ghidra/program/util/FunctionUtility.java @@ -19,7 +19,7 @@ import java.util.*; import generic.theme.GThemeDefaults.Colors.Palette; import ghidra.program.model.address.Address; -import ghidra.program.model.data.DataType; +import ghidra.program.model.data.*; import ghidra.program.model.lang.CompilerSpec; import ghidra.program.model.lang.Language; import ghidra.program.model.listing.*; @@ -50,7 +50,7 @@ public class FunctionUtility { * the target namespace. This probably can't happen */ public static void applyNameAndNamespace(Function target, Function source) - throws DuplicateNameException, InvalidInputException, CircularDependencyException { + throws DuplicateNameException, InvalidInputException, CircularDependencyException { String name = source.getName(); Namespace targetNamespace = getOrCreateSourceNamespaceInTarget(target, source); Symbol symbol = target.getSymbol(); @@ -61,24 +61,68 @@ public class FunctionUtility { * Updates the destination function so its signature will match the source function's signature * as closely as possible. This method will try to create conflict names if necessary for the * function and its parameters. + *
+ * All datatypes will be resolved using the + * {@link DataTypeConflictHandler#DEFAULT_HANDLER default conflict handler}. + * * @param destinationFunction the destination function to update * @param sourceFunction the source function to use as a template * @throws InvalidInputException if the function name or a variable name is invalid or if a - * parameter data type is not a fixed length. + * parameter data type is not a fixed length. * @throws DuplicateNameException This shouldn't happen since it will try to create conflict - * names for the function and its variables if necessary. Otherwise, this would be because - * the function's name or a variable name already exists. + * names for the function and its variables if necessary. Otherwise, + * this would be because the function's name or a variable name already exists. * @throws CircularDependencyException if namespaces have circular references */ public static void updateFunction(Function destinationFunction, Function sourceFunction) - throws InvalidInputException, DuplicateNameException, CircularDependencyException { + throws InvalidInputException, DuplicateNameException, CircularDependencyException { - updateFunctionExceptName(destinationFunction, sourceFunction); + applySignature(destinationFunction, sourceFunction, false, + DataTypeConflictHandler.DEFAULT_HANDLER); + } + + /** + * Updates the destination function so its signature will match the source function's signature + * as closely as possible. This method will try to create conflict names if necessary for the + * function and its parameters. + * + * @param destinationFunction the destination function to update + * @param sourceFunction the source function to use as a template + * @param applyEmptyComposites If true, applied composites will be resolved without their + * respective components if the type does not already exist in the + * destination datatype manager. If false, normal type resolution + * will occur. + * @param conflictHandler conflict handler to be used when applying datatypes to the + * destination program. If this value is not null or + * {@link DataTypeConflictHandler#DEFAULT_HANDLER} the datatypes will be + * resolved prior to updating the destinationFunction. This handler + * will provide some control over how applied datatype are handled when + * they conflict with existing datatypes. + * See {@link DataTypeConflictHandler} which provides some predefined + * handlers. + * @throws InvalidInputException if the function name or a variable name is invalid or if a + * parameter data type is not a fixed length. + * @throws DuplicateNameException This shouldn't happen since it will try to create conflict + * names for the function and its variables if necessary. Otherwise, + * this would be because the function's name or a variable name already exists. + * @throws CircularDependencyException if namespaces have circular references + */ + public static void applySignature(Function destinationFunction, Function sourceFunction, + boolean applyEmptyComposites, DataTypeConflictHandler conflictHandler) + throws InvalidInputException, DuplicateNameException, CircularDependencyException { + + if (conflictHandler == null) { + conflictHandler = DataTypeConflictHandler.DEFAULT_HANDLER; + } + updateFunctionExceptName(destinationFunction, sourceFunction, applyEmptyComposites, + conflictHandler); applyNameAndNamespace(destinationFunction, sourceFunction); } private static void updateFunctionExceptName(Function destinationFunction, - Function sourceFunction) throws InvalidInputException, DuplicateNameException { + Function sourceFunction, boolean applyEmptyComposites, + DataTypeConflictHandler conflictHandler) + throws InvalidInputException, DuplicateNameException { Program sourceProgram = sourceFunction.getProgram(); Program destinationProgram = destinationFunction.getProgram(); @@ -88,23 +132,47 @@ public class FunctionUtility { determineCallingConventionName(destinationFunction, sourceFunction, sameLanguage); boolean useCustomStorage = determineCustomStorageUse(destinationFunction, sourceFunction, sameLanguage); - boolean force = true; - SourceType source = sourceFunction.getSignatureSource(); - Variable returnValue = - determineReturnValue(destinationFunction, sourceFunction, useCustomStorage); - List newParams = - determineParameters(destinationFunction, sourceFunction, useCustomStorage); - setUniqueParameterNames(destinationFunction, newParams); - destinationFunction.updateFunction(callingConventionName, returnValue, newParams, - useCustomStorage ? FunctionUpdateType.CUSTOM_STORAGE - : FunctionUpdateType.DYNAMIC_STORAGE_ALL_PARAMS, - force, source); + + DataTypeManager destinationDtm = destinationFunction.getProgram().getDataTypeManager(); + final DataTypeCleaner dtCleaner = + applyEmptyComposites ? new DataTypeCleaner(destinationDtm, true) : null; + try { + SourceType source = sourceFunction.getSignatureSource(); + Variable returnValue = + determineReturnValue(destinationFunction, sourceFunction, useCustomStorage, + dt -> prepareDataType(dt, destinationDtm, dtCleaner, conflictHandler)); + List newParams = + determineParameters(destinationFunction, sourceFunction, useCustomStorage, + dt -> prepareDataType(dt, destinationDtm, dtCleaner, conflictHandler)); + setUniqueParameterNames(destinationFunction, newParams); + destinationFunction.updateFunction(callingConventionName, returnValue, newParams, + useCustomStorage ? FunctionUpdateType.CUSTOM_STORAGE + : FunctionUpdateType.DYNAMIC_STORAGE_ALL_PARAMS, + true, source); + } + finally { + if (dtCleaner != null) { + dtCleaner.close(); + } + } + applyInline(destinationFunction, sourceFunction); applyNoReturn(destinationFunction, sourceFunction); applyVarArgs(destinationFunction, sourceFunction); applyCallFixup(destinationFunction, sourceFunction, sameLanguage); } + private static DataType prepareDataType(DataType dt, DataTypeManager destinationDtm, + DataTypeCleaner dtCleaner, DataTypeConflictHandler conflictHandler) { + if (dtCleaner != null) { + dt = dtCleaner.clean(dt); + } + if (conflictHandler != DataTypeConflictHandler.DEFAULT_HANDLER) { + dt = destinationDtm.resolve(dt, conflictHandler); + } + return dt; + } + /** * Changes the names of the parameters in the array to unique names that won't conflict with * any other names in the function's namespace when the parameters are used to replace @@ -117,7 +185,7 @@ public class FunctionUtility { * @throws DuplicateNameException */ public static void setUniqueParameterNames(Function function, List parameters) - throws DuplicateNameException, InvalidInputException { + throws DuplicateNameException, InvalidInputException { SymbolTable symbolTable = function.getProgram().getSymbolTable(); // Create a set containing all the unique parameter names determined so far so they can @@ -146,7 +214,7 @@ public class FunctionUtility { * @return a unique parameter name */ private static String getUniqueReplacementParameterName(SymbolTable symbolTable, - Function function, String name, Set namesNotToBeUsed) { + Function function, String name, Set namesNotToBeUsed) { if (name == null || SymbolUtilities.isDefaultParameterName(name)) { return name; } @@ -166,7 +234,7 @@ public class FunctionUtility { * that doesn't conflict with any in the set of names not to be used. */ private static String getUniqueNameIgnoringCurrentParameters(SymbolTable symbolTable, - Function function, String baseName, Set namesNotToBeUsed) { + Function function, String baseName, Set namesNotToBeUsed) { String name = baseName; if (name != null) { // establish unique name @@ -211,7 +279,7 @@ public class FunctionUtility { } private static void applyCallFixup(Function destinationFunction, Function sourceFunction, - boolean sameLanguage) { + boolean sameLanguage) { String sourceCallFixup = sourceFunction.getCallFixup(); String destCallFixup = destinationFunction.getCallFixup(); if (SystemUtilities.isEqual(destCallFixup, sourceCallFixup)) { @@ -233,7 +301,7 @@ public class FunctionUtility { * already exists. */ static void setFunctionName(Function destinationFunction, Function sourceFunction) - throws InvalidInputException, DuplicateNameException { + throws InvalidInputException, DuplicateNameException { String sourceName = sourceFunction.getName(); Address sourceEntryPoint = sourceFunction.getEntryPoint(); Namespace sourceNamespace = sourceFunction.getParentNamespace(); @@ -307,13 +375,17 @@ public class FunctionUtility { } private static String determineCallingConventionName(Function destinationFunction, - Function sourceFunction, boolean sameLanguageAndCompilerSpec) { + Function sourceFunction, boolean sameLanguageAndCompilerSpec) { + String sourceConv = sourceFunction.getCallingConventionName(); + if (CompilerSpec.CALLING_CONVENTION_thiscall.equals(sourceConv)) { + return sourceConv; + } return sameLanguageAndCompilerSpec ? sourceFunction.getCallingConventionName() - : destinationFunction.getCallingConventionName(); + : destinationFunction.getCallingConventionName(); } private static boolean determineCustomStorageUse(Function destinationFunction, - Function sourceFunction, boolean sameLanguage) { + Function sourceFunction, boolean sameLanguage) { boolean useCustomStorage = sourceFunction.hasCustomVariableStorage(); if (useCustomStorage) { return sameLanguage; // only use for same language. @@ -322,28 +394,39 @@ public class FunctionUtility { } private static Variable determineReturnValue(Function destinationFunction, - Function sourceFunction, boolean useCustomStorage) throws InvalidInputException { + Function sourceFunction, boolean useCustomStorage, + java.util.function.Function prepareDataType) + throws InvalidInputException { Program destinationProgram = destinationFunction.getProgram(); Parameter sourceReturn = sourceFunction.getReturn(); - DataType dataType = sourceReturn.getDataType(); VariableStorage storage = (useCustomStorage) ? sourceReturn.getVariableStorage().clone(destinationProgram) - : VariableStorage.UNASSIGNED_STORAGE; + : VariableStorage.UNASSIGNED_STORAGE; + DataType dataType = prepareDataType.apply(sourceReturn.getDataType()); + if (dataType.isZeroLength()) { + storage = VariableStorage.UNASSIGNED_STORAGE; + } Parameter returnValue = new ReturnParameterImpl(dataType, storage, destinationProgram); return returnValue; } private static List determineParameters(Function destinationFunction, - Function sourceFunction, boolean useCustomStorage) throws InvalidInputException { + Function sourceFunction, boolean useCustomStorage, + java.util.function.Function prepareDataType) + throws InvalidInputException { Program destinationProgram = destinationFunction.getProgram(); List parameters = new ArrayList<>(); Parameter[] sourceParameters = sourceFunction.getParameters(); for (Parameter sourceParameter : sourceParameters) { String name = sourceParameter.getName(); - DataType dataType = sourceParameter.getDataType(); VariableStorage storage = (useCustomStorage) ? sourceParameter.getVariableStorage().clone(destinationProgram) - : VariableStorage.UNASSIGNED_STORAGE; + : VariableStorage.UNASSIGNED_STORAGE; + DataType dataType = prepareDataType.apply(sourceParameter.getDataType()); + if (dataType.isZeroLength()) { + storage = VariableStorage.UNASSIGNED_STORAGE; + } + SourceType source = sourceParameter.getSource(); Parameter parameter = new ParameterImpl(name, dataType, storage, destinationProgram, source); @@ -399,7 +482,7 @@ public class FunctionUtility { } private static Namespace getOrCreateSourceNamespaceInTarget(Function target, Function source) - throws DuplicateNameException, InvalidInputException { + throws DuplicateNameException, InvalidInputException { Namespace targetNamespace = target.getParentNamespace(); Namespace sourceNamespace = source.getParentNamespace(); @@ -410,7 +493,7 @@ public class FunctionUtility { } private static Namespace getOrCreateTargetNamespace(Program program, Namespace otherNamespace) - throws DuplicateNameException, InvalidInputException { + throws DuplicateNameException, InvalidInputException { if (otherNamespace.isGlobal()) { return program.getGlobalNamespace(); } diff --git a/Ghidra/Features/CodeCompare/Module.manifest b/Ghidra/Features/CodeCompare/Module.manifest new file mode 100755 index 0000000000..5f6a55051e --- /dev/null +++ b/Ghidra/Features/CodeCompare/Module.manifest @@ -0,0 +1 @@ +EXCLUDE FROM GHIDRA JAR: true diff --git a/Ghidra/Features/CodeCompare/build.gradle b/Ghidra/Features/CodeCompare/build.gradle new file mode 100755 index 0000000000..e98116eb04 --- /dev/null +++ b/Ghidra/Features/CodeCompare/build.gradle @@ -0,0 +1,26 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +apply from: "$rootProject.projectDir/gradle/distributableGhidraModule.gradle" +apply from: "$rootProject.projectDir/gradle/javaProject.gradle" +apply from: "$rootProject.projectDir/gradle/javaTestProject.gradle" + + +apply plugin: 'eclipse' +eclipse.project.name = 'Features CodeCompare' + +dependencies { + api project(":Decompiler") +} diff --git a/Ghidra/Features/CodeCompare/certification.manifest b/Ghidra/Features/CodeCompare/certification.manifest new file mode 100755 index 0000000000..8fae79a3bf --- /dev/null +++ b/Ghidra/Features/CodeCompare/certification.manifest @@ -0,0 +1,4 @@ +##VERSION: 2.0 +##MODULE IP: LGPL 3.0 +Module.manifest||GHIDRA||||END| +data/codecompare.theme.properties||GHIDRA||||END| diff --git a/Ghidra/Features/CodeCompare/data/codecompare.theme.properties b/Ghidra/Features/CodeCompare/data/codecompare.theme.properties new file mode 100644 index 0000000000..25cb691263 --- /dev/null +++ b/Ghidra/Features/CodeCompare/data/codecompare.theme.properties @@ -0,0 +1,16 @@ + +[Defaults] + +color.bg.codecompare.highlight.diff = cyan +color.bg.codecompare.highlight.field.diff.matching = pink +color.bg.codecompare.highlight.field.diff.not.matching = lavender +color.bg.codecompare.highlight.field.diff.other = orange + + + +[Dark Defaults] + +color.bg.codecompare.highlight.diff = #356493 +color.bg.codecompare.highlight.field.diff.matching = #994399 +color.bg.codecompare.highlight.field.diff.not.matching = #9775ff +color.bg.codecompare.highlight.field.diff.other = #936a17 \ No newline at end of file diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/AbstractMatchedTokensAction.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/AbstractMatchedTokensAction.java new file mode 100644 index 0000000000..a7cd616b77 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/AbstractMatchedTokensAction.java @@ -0,0 +1,136 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import java.util.Iterator; +import java.util.List; + +import docking.ActionContext; +import docking.action.DockingAction; +import ghidra.app.decompiler.ClangToken; +import ghidra.app.decompiler.DecompilerLocation; +import ghidra.app.decompiler.component.*; +import ghidra.codecompare.graphanalysis.TokenBin; +import ghidra.program.model.listing.Program; + +/** + * This is a base class for actions in a {@link DecompilerDiffCodeComparisonPanel} + */ +public abstract class AbstractMatchedTokensAction extends DockingAction { + + protected DecompilerDiffCodeComparisonPanel diffPanel; + protected boolean disableOnReadOnly; + + /** + * Constructor + * + * @param actionName name of action + * @param owner owner of action + * @param diffPanel diff panel containing action + * @param disableOnReadOnly if true, action will be disabled for read-only programs + */ + public AbstractMatchedTokensAction(String actionName, String owner, + DecompilerDiffCodeComparisonPanel diffPanel, boolean disableOnReadOnly) { + super(actionName, owner); + this.diffPanel = diffPanel; + this.disableOnReadOnly = disableOnReadOnly; + } + + /** + * Determines whether the action should be enable for a pair of + * matching tokens. + * + * @param tokenPair tokens + * @return true if action should be enabled + */ + protected abstract boolean enabledForTokens(TokenPair tokenPair); + + @Override + public boolean isEnabledForContext(ActionContext context) { + if (!(context instanceof DualDecompilerActionContext compareContext)) { + return false; + } + if (!(compareContext + .getCodeComparisonPanel() instanceof DecompilerCodeComparisonPanel decompPanel)) { + return false; + } + + if (disableOnReadOnly) { + //get the program corresponding to the panel with focus + Program program = decompPanel.getLeftProgram(); + if (program == null) { + return false; //panel initializing; don't enable action + } + if (!decompPanel.leftPanelHasFocus()) { + program = decompPanel.getRightProgram(); + } + if (!program.canSave()) { + return false; //program is read-only, don't enable action + } + } + + @SuppressWarnings("unchecked") + TokenPair currentPair = getCurrentTokenPair(decompPanel); + return enabledForTokens(currentPair); + + } + + /** + * Returns a {@link TokenPair} consisting of the token under the cursor in the focused + * decompiler panel and its counterpart in the other panel. + * + * @param decompPanel decomp panel + * @return matching tokens (or null if no match) + */ + protected TokenPair getCurrentTokenPair( + DecompilerCodeComparisonPanel decompPanel) { + + DecompilerPanel focusedPanel = decompPanel.getFocusedDecompilerPanel().getDecompilerPanel(); + + if (!(focusedPanel.getCurrentLocation() instanceof DecompilerLocation focusedLocation)) { + return null; + } + + ClangToken focusedToken = focusedLocation.getToken(); + if (focusedToken == null) { + return null; + } + List tokenBin = diffPanel.getHighBins(); + if (tokenBin == null) { + return null; + } + TokenBin containingBin = TokenBin.getBinContainingToken(tokenBin, focusedToken); + if (containingBin == null) { + return null; + } + TokenBin matchedBin = containingBin.getMatch(); + if (matchedBin == null) { + return null; + } + //loop over the tokens in the matching bin and return the first one in the same + //class as focusedToken + Iterator tokenIter = matchedBin.iterator(); + while (tokenIter.hasNext()) { + ClangToken currentMatch = tokenIter.next(); + if (currentMatch.getClass().equals(focusedToken.getClass())) { + return decompPanel.leftPanelHasFocus() ? new TokenPair(focusedToken, currentMatch) + : new TokenPair(currentMatch, focusedToken); + } + } + return null; + } + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/CodeDiffFieldPanelCoordinator.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/CodeDiffFieldPanelCoordinator.java new file mode 100755 index 0000000000..5c3023d913 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/CodeDiffFieldPanelCoordinator.java @@ -0,0 +1,304 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import java.math.BigInteger; +import java.util.ArrayList; +import java.util.List; + +import org.apache.commons.collections4.BidiMap; +import org.apache.commons.collections4.bidimap.DualHashBidiMap; + +import docking.widgets.fieldpanel.support.ViewerPosition; +import ghidra.app.decompiler.*; +import ghidra.app.decompiler.component.DecompilerPanel; +import ghidra.app.decompiler.component.DualDecompilerFieldPanelCoordinator; +import ghidra.codecompare.graphanalysis.TokenBin; +import ghidra.program.model.pcode.HighFunction; +import ghidra.program.util.ProgramLocation; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +/** + * Class to coordinate the scrolling of two decompiler panels as well as cursor location + * highlighting due to cursor location changes. + */ +public class CodeDiffFieldPanelCoordinator extends DualDecompilerFieldPanelCoordinator { + + private BidiMap leftToRightLineNumberPairing; + private List leftLines = new ArrayList(); + private List rightLines = new ArrayList(); + private int lockedLeftLineNumber = 0; + private int lockedRightLineNumber = 0; + private DecompilerPanel leftDecompilerPanel; + private DecompilerPanel rightDecompilerPanel; + private boolean matchConstantsExactly; + private List highBins; + + /** + * Constructor + * @param dualDecompilerPanel decomp comparison panel + */ + public CodeDiffFieldPanelCoordinator(DecompilerDiffCodeComparisonPanel dualDecompilerPanel) { + super(dualDecompilerPanel); + this.leftDecompilerPanel = dualDecompilerPanel.getLeftDecompilerPanel(); + this.rightDecompilerPanel = dualDecompilerPanel.getRightDecompilerPanel(); + leftToRightLineNumberPairing = new DualHashBidiMap<>(); + } + + /** + * Computes the line pairing for two decompiled functions. Any existing line pairing is + * cleared. + * + * @param decompileDataDiff decomp diff + * @param monitor monitor + * @throws CancelledException if user cancels + */ + public void computeLinePairing(DecompileDataDiff decompileDataDiff, TaskMonitor monitor) + throws CancelledException { + highBins = decompileDataDiff.getTokenMap(matchConstantsExactly, monitor); + HighFunction leftHighFunction = decompileDataDiff.getLeftHighFunction(); + + clearLineNumberPairing(); + for (TokenBin bin : highBins) { + if (bin.getMatch() != null) { + boolean isLeftBin = bin.getHighFunction().equals(leftHighFunction); + ClangToken binToken = bin.get(0); + ClangToken sidekickToken = bin.getMatch().get(0); + ClangToken leftClangToken = isLeftBin ? binToken : sidekickToken; + ClangToken rightClangToken = isLeftBin ? sidekickToken : binToken; + ClangLine leftLine = leftClangToken.getLineParent(); + ClangLine rightLine = rightClangToken.getLineParent(); + leftToRightLineNumberPairing.put(leftLine.getLineNumber(), + rightLine.getLineNumber()); + } + } + } + + @Override + public void leftLocationChanged(ProgramLocation leftLocation) { + DecompilerLocation leftDecompilerLocation = (DecompilerLocation) leftLocation; + + // Get the line from the token so we can match it with the line from the other panel. + ClangToken leftToken = + (leftDecompilerLocation != null) ? leftDecompilerLocation.getToken() : null; + ClangLine leftLine = (leftToken != null) ? leftToken.getLineParent() : null; + if (leftLine != null) { + int leftLineNumber = leftLine.getLineNumber(); + if (searchLeftForPair(leftLineNumber)) { + lockLines(BigInteger.valueOf(lockedLeftLineNumber), + BigInteger.valueOf(lockedRightLineNumber)); + } + } + panelViewChanged(leftDecompilerPanel); + } + + @Override + public void rightLocationChanged(ProgramLocation rightLocation) { + DecompilerLocation rightDecompilerLocation = (DecompilerLocation) rightLocation; + + // Get the line from the token so we can try to match it with the line from the other panel. + ClangToken rightToken = + (rightDecompilerLocation != null) ? rightDecompilerLocation.getToken() : null; + ClangLine rightLine = (rightToken != null) ? rightToken.getLineParent() : null; + if (rightLine != null) { + int rightLineNumber = rightLine.getLineNumber(); + if (searchRightForPair(rightLineNumber)) { + lockLines(BigInteger.valueOf(lockedLeftLineNumber), + BigInteger.valueOf(lockedRightLineNumber)); + } + } + panelViewChanged(rightDecompilerPanel); + } + + /** + * + * Updates the comparison panel using {@code decompileDataDiff} + * + * @param decompileDataDiff decomp diff data + * @param shouldMatchConstantsExactly if differences in constant values should count + * @param monitor monitor + * @throws CancelledException if user cancels + */ + public void replaceDecompileDataDiff(DecompileDataDiff decompileDataDiff, + boolean shouldMatchConstantsExactly, TaskMonitor monitor) throws CancelledException { + + this.matchConstantsExactly = shouldMatchConstantsExactly; + if (leftDecompilerPanel != null) { + leftLines = leftDecompilerPanel.getLines(); + } + else { + leftLines.clear(); + } + + if (rightDecompilerPanel != null) { + rightLines = rightDecompilerPanel.getLines(); + } + else { + rightLines.clear(); + } + lockFunctionSignatureLines(); + computeLinePairing(decompileDataDiff, monitor); + } + + /** + * Clears the existing line number pairing + */ + void clearLineNumberPairing() { + leftToRightLineNumberPairing.clear(); + } + + List getHighBins() { + return highBins; + } + + private void panelViewChanged(DecompilerPanel panel) { + ViewerPosition viewerPosition = panel.getViewerPosition(); + BigInteger index = viewerPosition.getIndex(); + int xOffset = viewerPosition.getXOffset(); + int yOffset = viewerPosition.getYOffset(); + viewChanged(panel.getFieldPanel(), index, xOffset, yOffset); + } + + //locks the two function signature lines - used when first displaying the comparison + private void lockFunctionSignatureLines() { + + ClangLine leftLine = getClangLine(leftLines, 0); + ClangLine rightLine = getClangLine(rightLines, 0); + + // If we can find both lines with function signatures then lock lines on them. + ClangLine leftFunctionLine = findFunctionSignatureLine(leftLines); + ClangLine rightFunctionLine = findFunctionSignatureLine(rightLines); + if (leftFunctionLine != null && rightFunctionLine != null) { + leftLine = leftFunctionLine; + rightLine = rightFunctionLine; + } + + if (leftLine != null && rightLine != null) { + setLockedLineNumbers(leftLine.getLineNumber(), rightLine.getLineNumber()); + lockLines(BigInteger.valueOf(lockedLeftLineNumber), + BigInteger.valueOf(lockedRightLineNumber)); + } + } + + private ClangLine findFunctionSignatureLine(List lines) { + for (ClangLine clangLine : lines) { + int numTokens = clangLine.getNumTokens(); + for (int i = 0; i < numTokens; i++) { + ClangToken token = clangLine.getToken(i); + if (token instanceof ClangFuncNameToken) { + return clangLine; + } + } + } + return null; + } + + /** + * Gets the indicated line number from the list after adjusting the line number when + * it falls outside the lower or upper limits of the array list. If there are no items + * in the array list then null is returned. + * @param lines the ordered array list of ClangLines. + * @param lineNumber the decompiler line number (1 based, not 0 based). + * @return the ClangLine for the indicated line number. Otherwise, null. + */ + private ClangLine getClangLine(List lines, int lineNumber) { + if (lines.isEmpty()) { + return null; + } + int size = lines.size(); + if (lineNumber < 1) { + lineNumber = 1; + } + if (lineNumber > size) { + lineNumber = size; + } + return lines.get(lineNumber - 1); + } + + private void setLockedLineNumbers(int leftLineNumber, int rightLineNumber) { + lockedLeftLineNumber = leftLineNumber; + lockedRightLineNumber = rightLineNumber; + } + + private boolean searchRightForPair(int rightLineNumber) { + if (setLeftFromRight(rightLineNumber)) { + return true; + } + int lastLine = rightLines.size(); + int previous = rightLineNumber - 1; + int next = rightLineNumber + 1; + while (previous > 0 || next <= lastLine) { + if (previous > 0) { + if (setLeftFromRight(previous)) { + return true; + } + previous--; + } + if (next <= lastLine) { + if (setLeftFromRight(next)) { + return true; + } + next++; + } + } + return false; + } + + private boolean setLeftFromRight(int rightLineNumber) { + Integer leftLineNumber = leftToRightLineNumberPairing.getKey(rightLineNumber); + if (leftLineNumber == null) { + return false; + } + setLockedLineNumbers(leftLineNumber, rightLineNumber); + return true; + } + + private boolean searchLeftForPair(int leftLineNumber) { + if (setRightFromLeft(leftLineNumber)) { + return true; + } + int lastLine = leftLines.size(); + int previous = leftLineNumber - 1; + int next = leftLineNumber + 1; + while (previous > 0 || next <= lastLine) { + if (previous > 0) { + if (setRightFromLeft(previous)) { + return true; + } + previous--; + } + if (next <= lastLine) { + if (setRightFromLeft(next)) { + return true; + } + next++; + } + } + return false; + } + + private boolean setRightFromLeft(int leftLineNumber) { + Integer rightLineNumber = leftToRightLineNumberPairing.get(leftLineNumber); + if (rightLineNumber == null) { + return false; + } + setLockedLineNumbers(leftLineNumber, rightLineNumber); + return true; + + } + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/CompareFuncsFromMatchedTokensAction.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/CompareFuncsFromMatchedTokensAction.java new file mode 100644 index 0000000000..b0dcbdb5fa --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/CompareFuncsFromMatchedTokensAction.java @@ -0,0 +1,122 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import docking.ActionContext; +import docking.action.MenuData; +import ghidra.app.decompiler.ClangFuncNameToken; +import ghidra.app.decompiler.component.DecompilerCodeComparisonPanel; +import ghidra.app.decompiler.component.DualDecompilerActionContext; +import ghidra.app.plugin.core.functioncompare.FunctionComparisonProvider; +import ghidra.app.services.FunctionComparisonService; +import ghidra.framework.plugintool.PluginTool; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.program.model.pcode.PcodeOp; +import ghidra.util.HelpLocation; +import ghidra.util.Msg; + +/** + * An action for bringing up a side-by-side function comparison of callees with matching + * tokens. + */ +public class CompareFuncsFromMatchedTokensAction extends AbstractMatchedTokensAction { + private PluginTool tool; + private static final String ACTION_NAME = "Compare Matching Callees"; + private static final String MENU_GROUP = "A1_Compare"; + private static final String HELP_TOPIC = "FunctionComparison"; + + /** + * Constructor + * @param diffPanel diff Panel + * @param tool tool + */ + public CompareFuncsFromMatchedTokensAction(DecompilerDiffCodeComparisonPanel diffPanel, + PluginTool tool) { + super(ACTION_NAME, tool.getName(), diffPanel, false); + this.tool = tool; + MenuData menuData = new MenuData(new String[] { ACTION_NAME }, null, MENU_GROUP); + setPopupMenuData(menuData); + setEnabled(true); + setHelpLocation(new HelpLocation(HELP_TOPIC, "Compare Matching Callees")); + } + + @Override + protected boolean enabledForTokens(TokenPair tokenPair) { + if (tokenPair == null) { + return false; + } + if (tokenPair.leftToken() == null || tokenPair.rightToken() == null) { + return false; + } + PcodeOp leftOp = tokenPair.leftToken().getPcodeOp(); + PcodeOp rightOp = tokenPair.rightToken().getPcodeOp(); + if (leftOp == null || rightOp == null) { + return false; + } + if (leftOp.getOpcode() != PcodeOp.CALL || rightOp.getOpcode() != PcodeOp.CALL) { + return false; + } + return (tokenPair.leftToken() instanceof ClangFuncNameToken) && + (tokenPair.rightToken() instanceof ClangFuncNameToken); + } + + @Override + public void actionPerformed(ActionContext context) { + if (!(context instanceof DualDecompilerActionContext compareContext)) { + return; + } + + if (!(compareContext + .getCodeComparisonPanel() instanceof DecompilerCodeComparisonPanel decompPanel)) { + return; + } + + @SuppressWarnings("unchecked") + TokenPair currentPair = getCurrentTokenPair(decompPanel); + if (currentPair == null || currentPair.leftToken() == null || + currentPair.rightToken() == null) { + return; + } + FunctionComparisonService service = tool.getService(FunctionComparisonService.class); + if (service == null) { + Msg.error(this, "Function Comparison Service not found!"); + return; + } + FunctionComparisonProvider comparisonProvider = service.createFunctionComparisonProvider(); + comparisonProvider.removeAddFunctionsAction(); + + ClangFuncNameToken leftFuncToken = (ClangFuncNameToken) currentPair.leftToken(); + ClangFuncNameToken rightFuncToken = (ClangFuncNameToken) currentPair.rightToken(); + + Function leftFunction = getFuncFromToken(leftFuncToken, decompPanel.getLeftProgram()); + Function rightFunction = getFuncFromToken(rightFuncToken, decompPanel.getRightProgram()); + + if (leftFunction == null || rightFunction == null) { + return; + } + + comparisonProvider.getModel().compareFunctions(leftFunction, rightFunction); + + } + + private Function getFuncFromToken(ClangFuncNameToken funcToken, Program program) { + Address callTarget = funcToken.getPcodeOp().getInput(0).getAddress(); + return program.getFunctionManager().getFunctionAt(callTarget); + } + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompileDataDiff.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompileDataDiff.java new file mode 100755 index 0000000000..56aa8807ed --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompileDataDiff.java @@ -0,0 +1,159 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import java.util.ArrayList; +import java.util.HashSet; + +import ghidra.app.decompiler.*; +import ghidra.app.decompiler.component.DecompileData; +import ghidra.codecompare.graphanalysis.Pinning; +import ghidra.codecompare.graphanalysis.TokenBin; +import ghidra.program.model.pcode.HighFunction; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +/** + * Class that takes decompile data for two functions (referred to as the left and right functions) + * and determines the differences between them. + */ +public class DecompileDataDiff { + + /** + * Pairing information for a single configuration of the Pinning algorithm, including: + * A list of all TokenBins across both functions that can be paired with each other + * (individual TokenBins are marked with their pair, if it exists). + * A list of tokens that do not have a match in the left function. + * A list of tokens that do not have a match in the right function. + */ + private static class Configuration { + ArrayList highBins; // List of all token bins + HashSet leftHighlightTokenSet; // Tokens without a match in the left function + HashSet rightHighlightTokenSet; // Tokens without a match in the right function + + public Configuration(ArrayList highBins, HighFunction leftFunc, + HighFunction rightFunc) { + this.highBins = highBins; + leftHighlightTokenSet = new HashSet<>(); + rightHighlightTokenSet = new HashSet<>(); + for (TokenBin bin : highBins) { + if (bin.getMatch() == null) { + for (ClangToken token : bin) { + ClangFunction clangFunction = token.getClangFunction(); + HighFunction highFunction = clangFunction.getHighFunction(); + if (leftFunc == highFunction) { + leftHighlightTokenSet.add(token); + } + else if (rightFunc == highFunction) { + rightHighlightTokenSet.add(token); + } + } + } + } + + } + } + + private DecompileData[] decompileData = new DecompileData[2]; + private ClangTokenGroup[] markup = new ClangTokenGroup[2]; + private HighFunction[] hfunc = new HighFunction[2]; + private boolean sizeCollapse; // True if we are comparing different size architectures + + // Different ways to configure the pinning algorithm + private static int NOT_EXACT_MATCH_CONSTANTS = 0; + private static int EXACT_MATCH_CONSTANTS = 1; + + private Configuration[] pairing; // Token pairing info from different Pinning configurations + + public DecompileDataDiff(DecompileData decompileData1, DecompileData decompileData2) { + this.decompileData[0] = decompileData1; + this.decompileData[1] = decompileData2; + int size1 = decompileData1.getProgram().getLanguage().getLanguageDescription().getSize(); + int size2 = decompileData2.getProgram().getLanguage().getLanguageDescription().getSize(); + sizeCollapse = (size1 != size2); + + markup[0] = decompileData[0].getCCodeMarkup(); + markup[1] = decompileData[1].getCCodeMarkup(); + + hfunc[0] = decompileData[0].getHighFunction(); + hfunc[1] = decompileData[1].getHighFunction(); + pairing = new Configuration[2]; + } + + /** + * Get sets of tokens (TokenBins) that have been paired across the two functions. + * The pairing can be performed either forcing constants to match, or not. + * @param matchConstantsExactly is true if constants should be forced to match + * @param monitor is the TaskMonitor + * @return the list of TokenBins + * @throws CancelledException if the user cancels the task + */ + public ArrayList getTokenMap(boolean matchConstantsExactly, TaskMonitor monitor) + throws CancelledException { + + int index = matchConstantsExactly ? EXACT_MATCH_CONSTANTS : NOT_EXACT_MATCH_CONSTANTS; + + if (pairing[index] == null) { + Pinning pin = Pinning.makePinning(hfunc[0], hfunc[1], matchConstantsExactly, + sizeCollapse, true, monitor); + ArrayList highBins = pin.buildTokenMap(markup[0], markup[1]); + pairing[index] = new Configuration(highBins, hfunc[0], hfunc[1]); + } + + return pairing[index].highBins; + } + + public HashSet getLeftHighlightTokenSet(boolean matchConstantsExactly, + TaskMonitor monitor) throws CancelledException { + + int index = matchConstantsExactly ? EXACT_MATCH_CONSTANTS : NOT_EXACT_MATCH_CONSTANTS; + + if (pairing[index] == null) { + getTokenMap(matchConstantsExactly, monitor); + } + + return pairing[index].leftHighlightTokenSet; + } + + public HashSet getRightHighlightTokenSet(boolean matchConstantsExactly, + TaskMonitor monitor) throws CancelledException { + + int matchConstantsIndex = + matchConstantsExactly ? EXACT_MATCH_CONSTANTS : NOT_EXACT_MATCH_CONSTANTS; + + if (pairing[matchConstantsIndex] == null) { + getTokenMap(matchConstantsExactly, monitor); + } + + return pairing[matchConstantsIndex].rightHighlightTokenSet; + } + + /** + * Gets the decompiled high level function for the left function. + * @return the left high level function + */ + public HighFunction getLeftHighFunction() { + return hfunc[0]; + } + + /** + * Gets the decompiled high level function for the right function. + * @return the right high level function + */ + public HighFunction getRightHighFunction() { + return hfunc[1]; + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompilerCodeComparisonOptions.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompilerCodeComparisonOptions.java new file mode 100644 index 0000000000..07d5a750b1 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompilerCodeComparisonOptions.java @@ -0,0 +1,139 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import java.awt.Color; + +import generic.theme.GColor; +import ghidra.framework.options.ToolOptions; +import ghidra.util.HelpLocation; + +/** + * This class holds the options for the decompiler diff view. + */ +public class DecompilerCodeComparisonOptions { + + private static final String MATCHING_TOKEN_HIGHLIGHT_KEY = "Focused Token Match Highlight"; + private static final String UNMATCHED_TOKEN_HIGHLIGHT_KEY = "Focused Token Unmatched Highlight"; + private static final String INELIGIBLE_TOKEN_HIGHLIGHT_KEY = + "Focused Token Ineligible Highlight"; + private static final String DIFF_HIGHLIGHT_KEY = "Difference Highlight"; + + private static final Color DEFAULT_MATCHING_TOKEN_HIGHLIGHT_COLOR = + new GColor("color.bg.codecompare.highlight.field.diff.matching"); + private static final Color DEFAULT_UNMATCHED_TOKEN_HIGHLIGHT_COLOR = + new GColor("color.bg.codecompare.highlight.field.diff.not.matching"); + private static final Color DEFAULT_INELIGIBLE_TOKEN_HIGHLIGHT_COLOR = + new GColor("color.bg.codecompare.highlight.field.diff.other"); + private static final Color DEFAULT_DIFF_HIGHLIGHT_COLOR = + new GColor("color.bg.codecompare.highlight.diff"); + + private static final String MATCHING_TOKEN_HIGHLIGHT_DESCRIPTION = + "Highlight Color for Focused Token and Match"; + private static final String UNMATCHED_TOKEN_HIGHLIGHT_DESCRIPTION = + "Highlight Color for a Focused Token with no Match"; + private static final String INELIGIBLE_TOKEN_HIGHLIGHT_DESCRIPTION = + "Highlight Color for a Focused Token which is ineligible for a match (e.g., whitespace)"; + private static final String DIFF_HIGHLIGHT_DESCRIPTION = "Highlight Color for Differences"; + + private Color matchingTokenHighlight; + private Color unmatchedTokenHighlight; + private Color ineligibleTokenHighlight; + private Color diffHighlight; + + public static final String OPTIONS_CATEGORY_NAME = "Decompiler Code Comparison"; + public static final String HELP_TOPIC = "FunctionComparison"; + + /** + * Constructor + */ + public DecompilerCodeComparisonOptions() { + + } + + /** + * Register the options + * @param options options + */ + public void registerOptions(ToolOptions options) { + HelpLocation help = new HelpLocation(HELP_TOPIC, "Decompiler Code Comparison Options"); + options.setOptionsHelpLocation(help); + + options.registerThemeColorBinding(MATCHING_TOKEN_HIGHLIGHT_KEY, + "color.bg.codecompare.highlight.field.diff.matching", help, + MATCHING_TOKEN_HIGHLIGHT_DESCRIPTION); + + options.registerThemeColorBinding(UNMATCHED_TOKEN_HIGHLIGHT_KEY, + "color.bg.codecompare.highlight.field.diff.not.matching", help, + UNMATCHED_TOKEN_HIGHLIGHT_DESCRIPTION); + + options.registerThemeColorBinding(INELIGIBLE_TOKEN_HIGHLIGHT_KEY, + "color.bg.codecompare.highlight.field.diff.other", help, + INELIGIBLE_TOKEN_HIGHLIGHT_DESCRIPTION); + + options.registerThemeColorBinding(DIFF_HIGHLIGHT_KEY, "color.bg.codecompare.highlight.diff", + help, DIFF_HIGHLIGHT_DESCRIPTION); + + } + + /** + * Read the options + * @param options options + */ + public void loadOptions(ToolOptions options) { + matchingTokenHighlight = + options.getColor(MATCHING_TOKEN_HIGHLIGHT_KEY, DEFAULT_MATCHING_TOKEN_HIGHLIGHT_COLOR); + unmatchedTokenHighlight = options.getColor(UNMATCHED_TOKEN_HIGHLIGHT_KEY, + DEFAULT_UNMATCHED_TOKEN_HIGHLIGHT_COLOR); + ineligibleTokenHighlight = options.getColor(INELIGIBLE_TOKEN_HIGHLIGHT_KEY, + DEFAULT_INELIGIBLE_TOKEN_HIGHLIGHT_COLOR); + diffHighlight = options.getColor(DIFF_HIGHLIGHT_KEY, DEFAULT_DIFF_HIGHLIGHT_COLOR); + } + + /** + * Returns the color used to highlight matches of the focused token + * @return match color + */ + public Color getFocusedTokenMatchHighlightColor() { + return matchingTokenHighlight; + } + + /** + * Returns the color used to highlight the focuses token when it does not have a match + * @return unmatched color + */ + public Color getFocusedTokenUnmatchedHighlightColor() { + return unmatchedTokenHighlight; + } + + /** + * Returns the color used to highlight the focused token when it is not eligible for a match + * (e.g., a whitespace token) + * @return ineligible color + */ + public Color getFocusedTokenIneligibleHighlightColor() { + return ineligibleTokenHighlight; + } + + /** + * Returns the color used to highlight differences between the two decompiled functions + * @return difference color + */ + public Color getDiffHighlightColor() { + return diffHighlight; + } + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompilerDiffCodeComparisonPanel.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompilerDiffCodeComparisonPanel.java new file mode 100755 index 0000000000..d48a1bebd0 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DecompilerDiffCodeComparisonPanel.java @@ -0,0 +1,252 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import java.util.List; + +import javax.swing.Icon; + +import docking.ActionContext; +import docking.action.*; +import docking.widgets.fieldpanel.internal.FieldPanelCoordinator; +import generic.theme.GIcon; +import ghidra.app.decompiler.component.*; +import ghidra.codecompare.graphanalysis.TokenBin; +import ghidra.framework.options.OptionsChangeListener; +import ghidra.framework.options.ToolOptions; +import ghidra.framework.plugintool.PluginTool; +import ghidra.util.HTMLUtilities; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.*; +import resources.Icons; +import resources.MultiIcon; + +/** + * This is a CodeComparisonPanel that gets discovered by other providers that display dual + * comparison views.
+ * Note: Therefore there may not be any other classes that refer directly to it. + */ +public class DecompilerDiffCodeComparisonPanel + extends DecompilerCodeComparisonPanel + implements DualDecompileResultsListener, OptionsChangeListener { + + public static final String CODE_DIFF_VIEW = "Decompiler Diff View"; + private DecompileDataDiff decompileDataDiff; + private DiffClangHighlightController leftHighlightController; + private DiffClangHighlightController rightHighlightController; + private CodeDiffFieldPanelCoordinator decompilerFieldPanelCoordinator; + private MyToggleExactConstantMatching toggleExactConstantMatchingAction; + private boolean isMatchingConstantsExactly = true; + private boolean toggleFlagWhenLastVisible = isMatchingConstantsExactly; + private CompareFuncsFromMatchedTokensAction compareFuncsAction; + private DecompilerCodeComparisonOptions comparisonOptions; + + /** + * Constructor + * @param owner owner + * @param tool tool + */ + public DecompilerDiffCodeComparisonPanel(String owner, PluginTool tool) { + super(owner, tool); + + comparisonOptions = new DecompilerCodeComparisonOptions(); + initializeOptions(); + leftHighlightController = new DiffClangHighlightController(comparisonOptions); + rightHighlightController = new DiffClangHighlightController(comparisonOptions); + setHighlightControllers(leftHighlightController, rightHighlightController); + + // Make the left highlight listen to the right. + leftHighlightController.addListener(rightHighlightController); + // Make the right highlight listen to the left. + rightHighlightController.addListener(leftHighlightController); + + addDualDecompileResultsListener(this); + decompilerFieldPanelCoordinator = new CodeDiffFieldPanelCoordinator(this); + setFieldPanelCoordinator(decompilerFieldPanelCoordinator); + + } + + private void initializeOptions() { + ToolOptions options = + tool.getOptions(DecompilerCodeComparisonOptions.OPTIONS_CATEGORY_NAME); + options.addOptionsChangeListener(this); + comparisonOptions.registerOptions(options); + comparisonOptions.loadOptions(options); + } + + @Override + public void optionsChanged(ToolOptions options, String optionName, Object oldValue, + Object newValue) { + comparisonOptions.loadOptions(options); + repaint(); + } + + @Override + public void setVisible(boolean aFlag) { + if (aFlag == isVisible()) { + return; + } + if (aFlag) { + // Becoming visible. + if (toggleFlagWhenLastVisible != isMatchingConstantsExactly) { + if (decompileDataDiff != null) { + determineDecompilerDifferences(); + } + } + } + else { + // No longer visible. + toggleFlagWhenLastVisible = isMatchingConstantsExactly; + } + super.setVisible(aFlag); + updateActionEnablement(); + } + + @Override + public String getTitle() { + return CODE_DIFF_VIEW; + } + + @Override + public void decompileResultsSet(DecompileData leftDecompileResults, + DecompileData rightDecompileResults) { + + if ((leftDecompileResults == null) || (rightDecompileResults == null) || + (leftDecompileResults.getFunction() == null) || + (rightDecompileResults.getFunction() == null)) { + return; + } + + decompileDataDiff = new DecompileDataDiff(leftDecompileResults, rightDecompileResults); + determineDecompilerDifferences(); + } + + List getHighBins() { + return decompilerFieldPanelCoordinator.getHighBins(); + } + + private void determineDecompilerDifferences() { + if (decompileDataDiff == null) { + return; + } + DetermineDecompilerDifferencesTask task = + new DetermineDecompilerDifferencesTask(decompileDataDiff, isMatchingConstantsExactly, + leftHighlightController, rightHighlightController, decompilerFieldPanelCoordinator); + + task.addTaskListener(new TaskListener() { + + @Override + public void taskCompleted(Task completedTask) { + // Does this need anything here? + } + + @Override + public void taskCancelled(Task cancelledTask) { + // Does this need anything here? + } + }); + + new TaskLauncher(task, getComponent()); + } + + @Override + protected void createActions() { + super.createActions(); + toggleExactConstantMatchingAction = new MyToggleExactConstantMatching(getClass().getName()); + compareFuncsAction = new CompareFuncsFromMatchedTokensAction(this, tool); + } + + @Override + public DockingAction[] getActions() { + DockingAction[] parentActions = super.getActions(); + DockingAction[] myActions = + new DockingAction[] { toggleExactConstantMatchingAction, compareFuncsAction }; + DockingAction[] allActions = new DockingAction[parentActions.length + myActions.length]; + System.arraycopy(parentActions, 0, allActions, 0, parentActions.length); + System.arraycopy(myActions, 0, allActions, parentActions.length, myActions.length); + return allActions; + } + + @Override + public void updateActionEnablement() { + // Need to enable/disable toolbar button. + toggleExactConstantMatchingAction.setEnabled(isVisible()); + } + + public class MyToggleExactConstantMatching extends ToggleDockingAction { + + private final Icon EXACT_CONSTANT_MATCHING_ICON = new GIcon("icon.base.source.c"); + private final Icon NO_EXACT_CONSTANT_MATCHING_ICON = + new MultiIcon(EXACT_CONSTANT_MATCHING_ICON, Icons.NOT_ALLOWED_ICON); + + /** + * Creates an action for toggling exact constant matching in the code diff's + * dual decompiler. + * @param owner the owner of this action (typically the provider). + */ + public MyToggleExactConstantMatching(String owner) { + super("Toggle Exact Constant Matching", owner); + + this.setToolBarData(new ToolBarData(NO_EXACT_CONSTANT_MATCHING_ICON, "toggles")); + + setDescription(HTMLUtilities.toHTML("Toggle whether or not constants must\n" + + "be exactly the same value to be a match\n" + "in the " + CODE_DIFF_VIEW + ".")); + setSelected(false); + setEnabled(true); + } + + @Override + public boolean isEnabledForContext(ActionContext context) { + if (context instanceof DualDecompilerActionContext) { + return true; + } + return false; + } + + @Override + public void actionPerformed(ActionContext context) { + isMatchingConstantsExactly = !isSelected(); + if (DecompilerDiffCodeComparisonPanel.this.isVisible()) { + DecompilerDiffCodeComparisonPanel.this.determineDecompilerDifferences(); + } + } + + @Override + public void setSelected(boolean selected) { + getToolBarData().setIcon( + selected ? NO_EXACT_CONSTANT_MATCHING_ICON : EXACT_CONSTANT_MATCHING_ICON); + super.setSelected(selected); + } + } + + @Override + protected CodeDiffFieldPanelCoordinator createFieldPanelCoordinator() { + CodeDiffFieldPanelCoordinator coordinator = new CodeDiffFieldPanelCoordinator(this); + if (decompileDataDiff != null) { + TaskBuilder.withRunnable(monitor -> { + try { + coordinator.replaceDecompileDataDiff(decompileDataDiff, + isMatchingConstantsExactly, monitor); + } + catch (CancelledException e) { + coordinator.clearLineNumberPairing(); + } + }).setTitle("Initializing Code Compare").launchNonModal(); + } + return coordinator; + + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DetermineDecompilerDifferencesTask.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DetermineDecompilerDifferencesTask.java new file mode 100755 index 0000000000..6520d029b7 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DetermineDecompilerDifferencesTask.java @@ -0,0 +1,79 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import java.util.List; +import java.util.Set; + +import ghidra.app.decompiler.ClangToken; +import ghidra.codecompare.graphanalysis.TokenBin; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.Task; +import ghidra.util.task.TaskMonitor; + +public class DetermineDecompilerDifferencesTask extends Task { + + private boolean matchConstantsExactly; + + private DiffClangHighlightController leftHighlightController; + private DiffClangHighlightController rightHighlightController; + + private DecompileDataDiff decompileDataDiff; + + private CodeDiffFieldPanelCoordinator decompilerFieldPanelCoordinator; + + public DetermineDecompilerDifferencesTask(DecompileDataDiff decompileDataDiff, + boolean matchConstantsExactly, DiffClangHighlightController leftHighlightController, + DiffClangHighlightController rightHighlightController, + CodeDiffFieldPanelCoordinator decompilerFieldPanelCoordinator) { + + super("Mapping C Tokens Between Functions", true, true, true); + this.decompileDataDiff = decompileDataDiff; + this.matchConstantsExactly = matchConstantsExactly; + this.leftHighlightController = leftHighlightController; + this.rightHighlightController = rightHighlightController; + this.decompilerFieldPanelCoordinator = decompilerFieldPanelCoordinator; + } + + @Override + public void run(TaskMonitor monitor) { + monitor.setMessage( + (matchConstantsExactly ? "Function Token Mapping By Matching Constants Exactly..." + : "Function Token Mapping WITHOUT Matching Constants Exactly...")); + try { + determineDifferences(monitor); + } + catch (CancelledException e) { + // User Cancelled. + } + } + + synchronized void determineDifferences(TaskMonitor monitor) throws CancelledException { + + List highBins = decompileDataDiff.getTokenMap(matchConstantsExactly, monitor); + Set leftHighlightTokenSet = + decompileDataDiff.getLeftHighlightTokenSet(matchConstantsExactly, monitor); + Set rightHighlightTokenSet = + decompileDataDiff.getRightHighlightTokenSet(matchConstantsExactly, monitor); + + leftHighlightController.setDiffHighlights(highBins, leftHighlightTokenSet); + rightHighlightController.setDiffHighlights(highBins, rightHighlightTokenSet); + + decompilerFieldPanelCoordinator.replaceDecompileDataDiff(decompileDataDiff, + matchConstantsExactly, monitor); + } + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DiffClangHighlightController.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DiffClangHighlightController.java new file mode 100755 index 0000000000..86616803be --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DiffClangHighlightController.java @@ -0,0 +1,242 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import java.awt.Color; +import java.util.*; +import java.util.stream.Collectors; + +import docking.widgets.EventTrigger; +import docking.widgets.fieldpanel.field.Field; +import docking.widgets.fieldpanel.support.FieldLocation; +import generic.theme.GColor; +import ghidra.app.decompiler.ClangSyntaxToken; +import ghidra.app.decompiler.ClangToken; +import ghidra.app.decompiler.component.*; +import ghidra.codecompare.graphanalysis.TokenBin; +import ghidra.util.ColorUtils; +import ghidra.util.SystemUtilities; +import util.CollectionUtils; + +/** + * Class to handle Function Difference highlights for a decompiled function. + */ + +public class DiffClangHighlightController extends LocationClangHighlightController + implements DiffClangHighlightListener { + + private Set diffTokenSet = new HashSet<>(); + private ClangToken locationToken; + private TokenBin locationTokenBin; + private List highlightBins; + private List listenerList = new ArrayList<>(); + private TokenBin matchingTokenBin; + private DecompilerCodeComparisonOptions comparisonOptions; + + public DiffClangHighlightController(DecompilerCodeComparisonOptions comparisonOptions) { + this.comparisonOptions = comparisonOptions; + } + + public void clearDiffHighlights() { + doClearDiffHighlights(); + notifyListeners(); + } + + private void doClearDiffHighlights() { + ClangToken[] array = diffTokenSet.toArray(new ClangToken[diffTokenSet.size()]); + for (ClangToken clangToken : array) { + clearDiffHighlight(clangToken); + } + } + + private void clearDiffHighlight(ClangToken clangToken) { + Color highlight = clangToken.getHighlight(); + if (highlight != null && highlight.equals(comparisonOptions.getDiffHighlightColor())) { + clangToken.setHighlight(null); + } + diffTokenSet.remove(clangToken); + } + + private void clearNonDiffHighlight(ClangToken clangToken) { + if (diffTokenSet.contains(clangToken)) { + clangToken.setHighlight(comparisonOptions.getDiffHighlightColor()); + } + else { + clangToken.setHighlight(null); + } + if (clangToken.isMatchingToken()) { + clangToken.setMatchingToken(false); + } + } + + public void setDiffHighlights(List highlightBins, Set tokenSet) { + this.highlightBins = highlightBins; + doClearDiffHighlights(); + for (ClangToken clangToken : tokenSet) { + clangToken.setHighlight(comparisonOptions.getDiffHighlightColor()); + diffTokenSet.add(clangToken); + } + notifyListeners(); + } + + @Override + public void fieldLocationChanged(FieldLocation location, Field field, EventTrigger trigger) { + + if (!(field instanceof ClangTextField)) { + return; + } + + // Get the token for the location so we can highlight its token bin. + // Also we will use it when notifying the other panel to highlight. + ClangToken tok = ((ClangTextField) field).getToken(location); + if (SystemUtilities.isEqual(locationToken, tok)) { + return; // Current location's token hasn't changed. + } + + // Undo any highlight of the previous matching tokenBin. + if (matchingTokenBin != null && matchingTokenBin.getMatch() != null) { + clearTokenBinHighlight(matchingTokenBin.getMatch()); + matchingTokenBin = null; + } + + clearCurrentLocationHighlight(); + + clearPrimaryHighlights(); + addPrimaryHighlight(tok, defaultHighlightColor); + if (tok instanceof ClangSyntaxToken) { + List tokens = addPrimaryHighlightToTokensForParenthesis( + (ClangSyntaxToken) tok, defaultParenColor); + reHighlightDiffs(tokens); + addBraceHighlight((ClangSyntaxToken) tok, defaultParenColor); + } + + TokenBin tokenBin = null; + if (tok != null) { + Color highlightColor = comparisonOptions.getFocusedTokenIneligibleHighlightColor(); // Don't know + if (highlightBins != null) { + tokenBin = TokenBin.getBinContainingToken(highlightBins, tok); + if (tokenBin != null) { + if (tokenBin.getMatch() != null) { + highlightColor = comparisonOptions.getFocusedTokenMatchHighlightColor(); + } + else if (tokenBin.getMatch() == null) { + highlightColor = comparisonOptions.getFocusedTokenUnmatchedHighlightColor(); + } + else { + // All the tokens that didn't fall into the "has a match" or "no match" + // categories above are in a single token bin. + // We don't want all these highlighted at the same time, so set the + // tokenBin to null. By doing this, only the current token gets highlighted. + tokenBin = null; + } + } + } + locationToken = tok; + locationTokenBin = tokenBin; + if (tokenBin == null) { + addPrimaryHighlight(tok, highlightColor); + } + else { + addTokenBinHighlight(tokenBin, highlightColor); + } + } + + // Notify other decompiler panel highlight controller we have a new location token. + for (DiffClangHighlightListener listener : listenerList) { + listener.locationTokenChanged(tok, tokenBin); + } + } + + private void reHighlightDiffs(List tokenList) { + Color averageColor = + ColorUtils.blend(defaultParenColor, comparisonOptions.getDiffHighlightColor(), 0.5); + for (ClangToken clangToken : tokenList) { + if (diffTokenSet.contains(clangToken)) { + clangToken.setHighlight(averageColor); + } + } + } + + private void clearCurrentLocationHighlight() { + if (locationTokenBin != null) { + clearTokenBinHighlight(locationTokenBin); + locationTokenBin = null; + locationToken = null; + } + if (locationToken != null) { + clearNonDiffHighlight(locationToken); + locationToken = null; + } + } + + private void addTokenBinHighlight(TokenBin tokenBin, Color highlightColor) { + for (ClangToken token : tokenBin) { + addPrimaryHighlight(token, highlightColor); + } + } + + private void clearTokenBinHighlight(TokenBin tokenBin) { + for (ClangToken token : tokenBin) { + clearNonDiffHighlight(token); + } + } + + private void doClearHighlights(TokenHighlights tokens) { + List clangTokens = + CollectionUtils.asStream(tokens).map(ht -> ht.getToken()).collect(Collectors.toList()); + for (ClangToken clangToken : clangTokens) { + clearNonDiffHighlight(clangToken); + } + tokens.clear(); + notifyListeners(); + } + + @Override + public void clearPrimaryHighlights() { + doClearHighlights(getPrimaryHighlights()); + } + + public boolean addListener(DiffClangHighlightListener listener) { + return listenerList.add(listener); + } + + public boolean removeListener(DiffClangHighlightListener listener) { + return listenerList.remove(listener); + } + + @Override + public void locationTokenChanged(ClangToken tok, TokenBin tokenBin) { + clearCurrentLocationHighlight(); + + // The token Changed in our other matching DiffClangHighlightController + highlightMatchingToken(tok, tokenBin); + } + + private void highlightMatchingToken(ClangToken tok, TokenBin tokenBin) { + // Undo any highlight of the previous matching tokenBin. + if (matchingTokenBin != null && matchingTokenBin.getMatch() != null) { + clearTokenBinHighlight(matchingTokenBin.getMatch()); + } + + // Highlight the new matching tokenBin. + if (tokenBin != null && tokenBin.getMatch() != null) { + addTokenBinHighlight(tokenBin.getMatch(), + comparisonOptions.getFocusedTokenMatchHighlightColor()); + } + + matchingTokenBin = tokenBin; + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DiffClangHighlightListener.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DiffClangHighlightListener.java new file mode 100755 index 0000000000..54092e85e1 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/DiffClangHighlightListener.java @@ -0,0 +1,30 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import ghidra.app.decompiler.ClangToken; +import ghidra.codecompare.graphanalysis.TokenBin; + +public interface DiffClangHighlightListener { + + /** + * Notifier that the current location changed to the specified token. + * @param tok the token + * @param tokenBin the bin which contains the token. Otherwise, null. + */ + public void locationTokenChanged(ClangToken tok, TokenBin tokenBin); + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/TokenPair.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/TokenPair.java new file mode 100644 index 0000000000..eab23c7eb0 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/TokenPair.java @@ -0,0 +1,21 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare; + +import ghidra.app.decompiler.ClangToken; + +record TokenPair(ClangToken leftToken, ClangToken rightToken) { +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlGraph.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlGraph.java new file mode 100755 index 0000000000..8cfe8006ec --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlGraph.java @@ -0,0 +1,107 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +import java.util.*; + +import ghidra.codecompare.graphanalysis.Pinning.Side; +import ghidra.program.model.pcode.HighFunction; +import ghidra.program.model.pcode.PcodeBlockBasic; + +/** + * A control-flow graph of a function for computing n-grams (CtrlNGram) that can be matched + * with another function. It mirrors the control-flow graph in HighFunction, but vertices, + * CtrlVertex, can have control-flow specific n-grams attached. + */ +public class CtrlGraph { + + Side side; // Which of the two functions being compared + HighFunction hfunc; // The control-flow graph from the decompiler being mirrored + Map blockToVertex; // Map from PcodeBlockBasic to corresponding CtrlVertex + ArrayList nodeList; // The list of nodes (basic blocks) in the graph + + /** + * Construct the control-flow graph, given a HighFunction. + * @param side is 0 or 1 indicating which of the two functions being compared + * @param hfunct is the decompiler produced HighFunction + */ + public CtrlGraph(Side side, HighFunction hfunct) { + this.side = side; + this.hfunc = hfunct; + this.nodeList = new ArrayList<>(); + this.blockToVertex = new HashMap<>(); + ArrayList blockList = this.hfunc.getBasicBlocks(); + + // Create the vertices from basic blocks + int uidCounter = 0; + for (PcodeBlockBasic curBlock : blockList) { + CtrlVertex temp = new CtrlVertex(curBlock, uidCounter++, this); + this.blockToVertex.put(curBlock, temp); + this.nodeList.add(temp); + } + + // Make the edges of the graph + for (PcodeBlockBasic curBlock : blockList) { + CtrlVertex curVert = this.blockToVertex.get(curBlock); + for (int i = 0; i < curBlock.getOutSize(); i++) { + PcodeBlockBasic neighborBlock = (PcodeBlockBasic) curBlock.getOut(i); + CtrlVertex neighborVert = this.blockToVertex.get(neighborBlock); + neighborVert.sources.add(curVert); + curVert.sinks.add(neighborVert); + } + } + } + + /** + * For every node in the graph, clear calculated n-grams. + */ + public void clearNGrams() { + for (CtrlVertex node : nodeList) { + node.clearNGrams(); + } + } + + /** + * Add extra distinguishing color to the 0-gram of each control-flow vertex. + */ + public void addEdgeColor() { + for (CtrlVertex vert : nodeList) { + vert.addEdgeColor(); + } + } + + /** + * Populate n-gram lists for every node. We generate two types of n-grams. One walking + * back from the root through sources, and the other walking forward from the root through sinks. + * @param numNGrams is the number of n-grams to generate per node + */ + public void makeNGrams(int numNGrams) { + int sourceSize = (numNGrams - 1) / 2 + 1; + for (int i = 0; i < sourceSize; ++i) { + for (CtrlVertex node : nodeList) { + node.nextNGramSource(i); // Construct (n+1)-gram from existing n-gram + } + } + for (CtrlVertex node : nodeList) { + node.nextNGramSink(0); // Produces index = (sourceSize + 1) + } + for (int i = sourceSize + 1; i < numNGrams - 1; ++i) { + for (CtrlVertex node : nodeList) { + node.nextNGramSink(i); + } + } + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlNGram.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlNGram.java new file mode 100755 index 0000000000..0a14e41c28 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlNGram.java @@ -0,0 +1,71 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +/** + * N-gram hash on the control-flow graph rooted at specific node {@link CtrlVertex}. + * The n-gram depth is the maximum number of (backward) edge traversals from the root + * node to any other node involved in the hash. The n-gram weight is the total number of + * nodes involved in the hash. The n-gram sorts with bigger weights first so that + * n-grams involving more nodes are paired first. + */ +public class CtrlNGram { + int weight; // The number of nodes involved in this hash + int depth; // The maximum distance between nodes in this n-gram set + int hash; // The hash + CtrlVertex root; // The root node of the n-gram + + /** + * Construct a control-flow n-gram. + * @param node is the root control-flow node from which the n-gram is computed + * @param weight is the number of nodes involved in computing the n-gram + * @param depth is the maximum distance between nodes involved in the n-gram + * @param hash is the hash value for the n-gram + */ + public CtrlNGram(CtrlVertex node, int weight, int depth, int hash) { + this.depth = depth; + this.weight = weight; + this.hash = hash; + this.root = node; + } + + /** + * Compare the hash of this n-gram with another. The weight and depth of the hashes must also + * be equal. The node(s) underlying the n-gram may be different. + * @param other is the other n-gram + * @return true if the hashes are the same + */ + public boolean equalHash(CtrlNGram other) { + if (other == null) { + return false; + } + // Compare just the hash data + if (weight == other.weight && depth == other.depth && hash == other.hash) { + return true; + } + return false; + } + + /** + * Check if this and another n-gram are rooted in different control-flow graphs + * @param other is the other n-gram to compare + * @return true if the n-grams are from different graphs + */ + public boolean graphsDiffer(CtrlNGram other) { + return (root.graph.side != other.root.graph.side); + } + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlVertex.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlVertex.java new file mode 100755 index 0000000000..47013b2325 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/CtrlVertex.java @@ -0,0 +1,147 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +import java.util.ArrayList; + +import ghidra.program.model.pcode.PcodeBlockBasic; + +/** + * A basic block in the control-flow graph of a function as produced by the Decompiler. + * A node stores references to immediate incoming nodes (sources). The node also stores hashes of n-grams + * {@link CtrlNGram} involving this node, where an n-gram is a set of adjacent nodes out to a depth of n. + */ +public class CtrlVertex { + private PcodeBlockBasic block; // Underlying basic block from the decompiler + int uid; // A unique id + ArrayList sources; // List of blocks flowing into this block + ArrayList sinks; // List of blocks this block flows into + ArrayList ngrams; // A list of n-grams (hashes of nearest neighbors) + CtrlGraph graph; // The control-flow graph owning this node + + /** + * Construct a control-flow vertex from a basic block. + * @param blk is the basic block + * @param id is a unique id to assign to the node + * @param grph is the graph that the node is part of + */ + public CtrlVertex(PcodeBlockBasic blk, int id, CtrlGraph grph) { + this.block = blk; + this.uid = id * 2 + grph.side.getValue(); + this.sources = new ArrayList<>(); + this.sinks = new ArrayList<>(); + this.ngrams = new ArrayList<>(); + int hash = depthZeroHash(0); + ngrams.add(new CtrlNGram(this, 1, 0, hash)); + this.graph = grph; + } + + /** + * Compute a hash of the meta-data associated with the node, not including edges. + * This is effectively a 0-gram of the node. It hashes info about the number of in + * and out edges to this node, but nothing else specific about neighbors. + * @param flavor is an extra context specific value to be included in the hash + * @return the hash + */ + int depthZeroHash(int flavor) { + int encoding = 1; // Initialize + encoding = Pinning.hashTwo(encoding, block.getInSize()); + encoding = Pinning.hashTwo(encoding, block.getOutSize()); + encoding = Pinning.hashTwo(encoding, flavor); + return encoding; + } + + /** + * Compute and store a new n-gram by combining existing (n-1)-grams from sources. + * @param index is the index of the current, already computed, (n-1)-gram to recurse on + */ + void nextNGramSource(int index) { + int nextSize = 1; + int masterSourceHash = 0; + for (CtrlVertex neighbor : sources) { // Assemble hashes from sources + CtrlNGram gram = neighbor.ngrams.get(index); + masterSourceHash += gram.hash; // Combine neighbors (n-1)-gram + nextSize += gram.weight; // Running tally of the number of nodes in the hash + } + + int nextEntry = ngrams.get(0).hash; + nextEntry = Pinning.hashTwo(nextEntry, masterSourceHash); + ngrams.add(new CtrlNGram(this, nextSize, ngrams.get(index).depth + 1, nextEntry)); + } + + /** + * Compute and store a new n-gram by combining existing (n-1)-grams from sinks. + * @param index is the index of the current, already computed, (n-1)-gram to recurse on + */ + void nextNGramSink(int index) { + int nextSize = 1; + int masterSourceHash = 0xfabcd; // Starting value to distinguish sinks + for (CtrlVertex neighbor : sinks) { // Assemble hashes from sources + CtrlNGram gram = neighbor.ngrams.get(index); + masterSourceHash += gram.hash; // Combine neighbors (n-1)-gram + nextSize += gram.weight; // Running tally of the number of nodes in the hash + } + + int nextEntry = ngrams.get(0).hash; + nextEntry = Pinning.hashTwo(nextEntry, masterSourceHash); + ngrams.add(new CtrlNGram(this, nextSize, ngrams.get(index).depth + 1, nextEntry)); + } + + /** + * Add some additional color to the 0-gram hash for this node. + * If the node has exactly 1 incoming edge, hash in the index of that edge, + * i.e. the position of that edge within the last of sink edges of the parent vertex. + * This distinguishes the node as either the true or false action after a conditional branch, or + * by associating the node with a particular case of a switch branch. + */ + void addEdgeColor() { + if (sources.size() == 1) { + CtrlNGram zeroGram = ngrams.get(0); + CtrlVertex src = sources.get(0); + int edgeColor; + for (edgeColor = 0; edgeColor < src.sinks.size(); ++edgeColor) { + if (src.sinks.get(edgeColor) == this) { + break; + } + } + zeroGram.hash = Pinning.hashTwo(zeroGram.hash, edgeColor); + } + } + + /** + * Remove everything except the 0-gram + */ + public void clearNGrams() { + while (ngrams.size() > 1) { + ngrams.remove(ngrams.size() - 1); + } + } + + /** + * Recompute the 0-gram, adding some additional salt to the hash + * @param flavor is the salt value to add + */ + public void setZeroGram(int flavor) { + int hash = depthZeroHash(flavor); + ngrams.get(0).hash = hash; + } + + @Override + public String toString() { + String result = Integer.toString(uid); + return result; + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataGraph.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataGraph.java new file mode 100755 index 0000000000..21408bf27f --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataGraph.java @@ -0,0 +1,345 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; + +import ghidra.codecompare.graphanalysis.Pinning.Side; +import ghidra.program.model.pcode.*; + +/** + * A data-flow graph of a function for computing n-grams {@link DataNGram} that can be matched + * with another function. The graph mirrors the HighFunction data-flow graph but unifies + * a Varnode and PcodeOp into a single node type (DataVertex) that can have n-grams attached. + * This graph can be modified relative to HighFunction to facilitate matching. + */ +public class DataGraph { + + /** + * Helper class for associating a DataVertex with another DataVertex. + * To distinguish multiple things associated with one DataVertex, an optional + * slot indicates through which input slot the association is made. + */ + public static class Associate { + DataVertex node; // The vertex having associations + int slot; // The input slot through which the association is made + + public Associate(DataVertex n, int sl) { + node = n; + slot = sl; + } + + @Override + public boolean equals(Object obj) { + if (!(obj instanceof Associate)) { + return false; + } + Associate other = (Associate) obj; + if (node.uid != other.node.uid) { + return false; + } + if (slot != other.slot) { + return false; + } + return true; + } + + @Override + public int hashCode() { + int val = node.hashCode(); + val = val * 31 + slot; + return val; + } + } + + Side side; // Which of two functions being compared + HighFunction hfunc; // The data-flow graph from the decompiler being mirrored + ArrayList nodeList; // Vertices in this graph + Map opToVert; // Map from PcodeOps to corresponding DataVertex + Map vnToVert; // Map from Varnodes to corresponding DataVertex + Map> associates; // Vertices that should get matched together + boolean constCaring; // True if constant values are factored into n-gram hashes + boolean ramCaring; // True if local/global is factored into n-gram hashes + boolean sizeCollapse; // True if big Varnodes are hashed as if they were 4 bytes + int pointerSize; // Number of bytes in a (default) pointer + long pointerMin; // Minimum offset to consider a pointer + + /** + * Construct the data-flow graph, given a HighFunction. Configuration parameters for later + * n-gram generation is given. + * @param side is 0 or 1 indicating which of the two functions being compared + * @param hfunc is the decompiler produced HighFunction + * @param constCaring is true if n-grams should take into account exact constant values + * @param ramCaring is true if n-grams should distinguish between local and global variables + * @param castCollapse is true if CAST operations should not be included in n-gram calculations + * @param sizeCollapse is true if variable sizes larger than 4 should be treated as size 4 + */ + public DataGraph(Side side, HighFunction hfunc, boolean constCaring, boolean ramCaring, + boolean castCollapse, boolean sizeCollapse) { + this.side = side; + this.constCaring = constCaring; + this.ramCaring = ramCaring; + this.sizeCollapse = sizeCollapse; + + pointerSize = hfunc.getFunction().getProgram().getDefaultPointerSize(); + pointerMin = (pointerSize < 3) ? 0xff : 0xffff; + ArrayList ptrsubs = new ArrayList<>(); + ArrayList casts = new ArrayList<>(); + + //Initialize the data inside. + this.nodeList = new ArrayList<>(); + this.hfunc = hfunc; + this.opToVert = new HashMap<>(); + this.vnToVert = new HashMap<>(); + this.associates = new HashMap<>(); + + int uidCounter = 0; // Counter for assigning unique ids + // Bring in all the Varnodes as vertices + Iterator vnIter = this.hfunc.locRange(); + while (vnIter.hasNext()) { + VarnodeAST currentVn = vnIter.next(); + if (currentVn.getDef() == null) { + if (currentVn.hasNoDescend()) { + continue; + } + } + DataVertex temp = new DataVertex(currentVn, this, uidCounter++); + this.nodeList.add(temp); + this.vnToVert.put(currentVn, temp); + } + // Bring in all the PcodeOps as vertices + Iterator opIter = this.hfunc.getPcodeOps(); + while (opIter.hasNext()) { + PcodeOpAST currentOp = opIter.next(); + DataVertex temp = new DataVertex(currentOp, this, uidCounter++); + this.nodeList.add(temp); + this.opToVert.put(currentOp, temp); + if (currentOp.getOpcode() == PcodeOp.PTRSUB) { + ptrsubs.add(temp); + } + if (currentOp.getOpcode() == PcodeOp.CAST) { + casts.add(temp); + } + } + + // Add edges to graph + opIter = this.hfunc.getPcodeOps(); + while (opIter.hasNext()) { + PcodeOpAST op = opIter.next(); + DataVertex node = this.opToVert.get(op); + for (int i = 0; i < op.getNumInputs(); i++) { + DataVertex sourceNode = this.vnToVert.get(op.getInput(i)); + if (sourceNode != null) { + node.sources.add(sourceNode); + sourceNode.sinks.add(node); + } + } + DataVertex sinkNode = this.vnToVert.get(op.getOutput()); + if (sinkNode != null) { + node.sinks.add(sinkNode); + sinkNode.sources.add(node); + } + } + + eliminatePtrsubs(ptrsubs); + + if (castCollapse) { + eliminateCasts(casts); + } + } + + /** + * @return the HighFunction this data-flow graph was generated from + */ + public HighFunction getHighFunction() { + return hfunc; + } + + /** + * Determine if the given Varnode is a constant and not a pointer. + * A constant is only considered a pointer if it has the size of a pointer and + * the constant value is not too "small". + * @param vn is the Varnode to check + * @return true if the constant is constant and not a pointer + */ + public boolean isConstantNonPointer(Varnode vn) { + if (!vn.isConstant()) { + return false; + } + if (vn.getSize() != pointerSize) { + return true; + } + long off = vn.getOffset(); + return (off >= 0 && off <= pointerMin); + } + + /** + * If a PTRSUB operation represents a &DAT_#, its input (a constant) is propagated forward + * to everything reading the PTRSUB, and the PTRSUB is eliminated. + * @param ptrsubs is the list of PTRSUB vertices + */ + private void eliminatePtrsubs(ArrayList ptrsubs) { + for (DataVertex subop : ptrsubs) { + DataVertex in0Node = subop.sources.get(0); + if (in0Node.vn.isConstant() && (in0Node.vn.getOffset() == 0)) { + DataVertex in1Node = subop.sources.get(1); + DataVertex outNode = subop.sinks.get(0); + in1Node.sinks.clear(); + replaceNodeInOutEdges(outNode, in1Node); + in0Node.clearEdges(); + subop.clearEdges(); + outNode.clearEdges(); + makeAssociation(subop, outNode, in1Node, 0); // Attach subop and outNode -> in1Node + } + } + } + + /** + * CAST operations are isolated in the graph and either: + * - The input replaces reads of the output, and the output is eliminated, OR + * - The output is redefined by defining PcodeOp op of the input, and the input is eliminated. + * @param casts is the list of CAST vertices + */ + private void eliminateCasts(ArrayList casts) { + for (DataVertex castNode : casts) { + + DataVertex in = castNode.sources.get(0); + DataVertex out = castNode.sinks.get(0); + DataVertex assoc = null; + int assocSlot = 0; + if (out.sinks.size() == 1) { + assoc = out.sinks.get(0); // Preferred node to associate with is the reading op + // Generate distinguishing slot for associate based on input slot CAST feeds into + for (assocSlot = 0; assocSlot < assoc.sources.size(); ++assocSlot) { + if (assoc.sources.get(assocSlot) == out) { + break; + } + } + } + + boolean outCast = true; + if ((out.sinks.size() == 1 && out.vn.isUnique()) || in.sources.size() == 0) { + outCast = false; + } + + if (outCast) { + // PcodeOp defining CAST input, now defines CAST output + // input is isolated + DataVertex topOp = in.sources.get(0); + topOp.sinks.clear(); + out.sources.clear(); + topOp.sinks.add(out); + out.sources.add(topOp); + in.clearEdges(); + if (assoc == null) { + assoc = out; + } + makeAssociation(castNode, in, assoc, assocSlot); + } + else { + // CAST input becomes input to descendants of CAST output + // output is isolated + removeInEdge(castNode, 0); + replaceNodeInOutEdges(out, in); + out.clearEdges(); + if (assoc == null) { + assoc = in; + } + makeAssociation(castNode, out, assoc, assocSlot); + } + castNode.clearEdges(); + } + } + + /** + * Populate n-gram lists for every node. + * @param numNGrams is the number of n-grams to generate per node + */ + public void makeNGrams(int numNGrams) { + for (int i = 0; i < numNGrams - 1; ++i) { + for (DataVertex node : nodeList) { + node.nextNGramSource(i); // Construct (n+1)-gram from existing n-gram + } + } + } + + /** + * Make an association between an (op,var) node pair that has been removed from the graph, with + * a node that remains in the graph. + * @param op is the operation node being removed + * @param var is the variable node being removed + * @param assoc is the node to associate with + * @param assocSlot is other distinguishing info about the association (incoming slot) + */ + private void makeAssociation(DataVertex op, DataVertex var, DataVertex assoc, int assocSlot) { + Associate key = new Associate(assoc, assocSlot); + ArrayList assocList = associates.get(key); + if (assocList == null) { + assocList = new ArrayList<>(); + associates.put(key, assocList); + } + assocList.add(op); + assocList.add(var); + } + + /** + * All out edges of the given node, become out edges of a replacement node. + * @param node is the given node + * @param replacement is the node receiving the new out edges + */ + private void replaceNodeInOutEdges(DataVertex node, DataVertex replacement) { + for (DataVertex outNode : node.sinks) { + for (int i = 0; i < outNode.sources.size(); i++) { + if (outNode.sources.get(i) == node) { + outNode.sources.set(i, replacement); + } + } + replacement.sinks.add(outNode); + } + node.sinks = new ArrayList<>(); + } + + /** + * Remove an edge between the given node and one of its inputs. + * @param node is the given node + * @param inEdge is the input edge + */ + private void removeInEdge(DataVertex node, int inEdge) { + DataVertex inNode = node.sources.get(inEdge); + int outEdge; + for (outEdge = 0; outEdge < inNode.sinks.size(); ++outEdge) { + if (inNode.sinks.get(outEdge) == node) { + break; + } + } + node.sources.remove(inEdge); + inNode.sinks.remove(outEdge); + } + + /** + * Dump a string representation of the data-flow graph. + * @param writer is the stream to write the string to + * @throws IOException for problems with the stream + */ + public void dump(Writer writer) throws IOException { + for (DataVertex vertex : nodeList) { + writer.append(vertex.toString()); + writer.append('\n'); + } + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataNGram.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataNGram.java new file mode 100755 index 0000000000..77967325bf --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataNGram.java @@ -0,0 +1,118 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +/** + * Sortable n-gram hash on the data-flow graph rooted at specific node {@link DataVertex}. + * The n-gram depth is the maximum number of (backward) edge traversals from the root + * node to any other node involved in the hash. The n-gram weight is the total number of + * nodes involved in the hash. The n-gram sorts with bigger weights first so that + * n-grams involving more nodes are paired first. + */ +public class DataNGram implements Comparable { + int weight; // The number of nodes involved in this hash + int depth; // The maximum distance between nodes in this n-gram set + int hash; // The hash + DataVertex root; // The root node of the n-gram + + /** + * Construct a data-flow n-gram. + * @param node is the root data-flow node from which the n-gram is computed + * @param weight is the number of data-flow nodes involved in the n-gram + * @param depth is the maximum distance between nodes involved in the n-gram + * @param hash is the hash value for the n-gram + */ + public DataNGram(DataVertex node, int weight, int depth, int hash) { + this.depth = depth; + this.weight = weight; + this.hash = hash; + this.root = node; + } + + @Override + public int compareTo(DataNGram other) { + if (this.weight > other.weight) { // Sort so that bigger weights come first + return -1; + } + else if (this.weight < other.weight) { + return 1; + } + + if (this.depth > other.depth) { // Sort on depth + return -1; + } + else if (this.depth < other.depth) { + return 1; + } + + if (this.hash > other.hash) { // Then sort on hash + return -1; + } + else if (this.hash < other.hash) { + return 1; + } + + if (this.root.uid > other.root.uid) { // For equivalent hashes, sort based on the node id + return -1; + } + else if (this.root.uid < other.root.uid) { + return 1; + } + + // Finally, sort on the graph owning the root node + return other.root.graph.side.compareTo(this.root.graph.side); + } + + /** + * Compare the hash of this n-gram with another. The weight and depth of the hashes must also + * be equal. The node(s) underlying the n-gram may be different. + * @param other is the other n-gram + * @return true if the hashes are the same + */ + public boolean equalHash(DataNGram other) { + if (other == null) { + return false; + } + // Compare just hash data + if (this.weight == other.weight && this.depth == other.depth && this.hash == other.hash) { + return true; + } + return false; + } + + /** + * Check if this and another n-gram are rooted in different data-flow graphs + * @param other is the other n-gram to compare + * @return true if the n-grams are from different graphs + */ + public boolean graphsDiffer(DataNGram other) { + if (this.root.graph != other.root.graph) { + return true; + } + return false; + } + + @Override + public String toString() { + StringBuilder buffer = new StringBuilder(); + buffer.append("d=").append(depth); + buffer.append(" h=").append(hash); + buffer.append(" w=").append(weight); + buffer.append(" vert=").append(root.uid); + return buffer.toString(); + } + +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataVertex.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataVertex.java new file mode 100755 index 0000000000..61fcf97b82 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/DataVertex.java @@ -0,0 +1,205 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +import java.util.ArrayList; + +import ghidra.program.model.pcode.*; + +/** + * A node in the data-flow graph of a function as produced by the Decompiler, represented by EITHER + * a Varnode or PcodeOp. A node stores references to immediate incoming nodes (sources) and immediate + * outgoing nodes (sinks). The node also stores hashes of n-grams involving this node, where + * an n-gram is a set of adjacent nodes out to a depth of n. + */ +public class DataVertex { + + int uid; // Unique identifying integer + PcodeOpAST op; // The underlying PcodeOp (or null) + VarnodeAST vn; // The underlying Varnode (or null) + DataGraph graph; // The containing graph + ArrayList sources; // Nodes with an incoming edge + ArrayList sinks; // Nodes with an outgoing edge + ArrayList ngrams; // A list of n-grams (hashes of nearest neighbors) rooted at this node + int passComplete; // Last pass for which this node was evaluated + boolean paired; // Found a match for this node + + /** + * Construct node from a PcodeOp + * @param myOp is the PcodeOp + * @param myGraph is the graph owning the node + * @param uniqueID is a unique id to assign to the node + */ + public DataVertex(PcodeOpAST myOp, DataGraph myGraph, int uniqueID) { + op = myOp; + vn = null; + commonConstructor(myGraph, uniqueID); + } + + /** + * Construct node from a Varnode + * @param myVn is the Varnode + * @param myGraph is the graph owning the node + * @param uniqueID is a unique id to assign to the node + */ + public DataVertex(VarnodeAST myVn, DataGraph myGraph, int uniqueID) { + vn = myVn; + op = null; + commonConstructor(myGraph, uniqueID); + } + + /** + * Initialize internals of the node. Allocate storage for edges and n-grams. + * @param myGraph is the graph owning the node + * @param uniqueID is a unique id to assign to the node + */ + private void commonConstructor(DataGraph myGraph, int uniqueID) { + uid = uniqueID * 2 + myGraph.side.getValue(); + graph = myGraph; + sources = new ArrayList(); + sinks = new ArrayList(); + ngrams = new ArrayList(); + int hash = depthZeroHash(); + ngrams.add(new DataNGram(this, 1, 0, hash)); + paired = false; + passComplete = -1; + } + + @Override + public String toString() { + StringBuilder buffer = new StringBuilder(); + buffer.append("uid=").append(uid).append(' '); + if (op != null) { + for (int i = 0; i < sinks.size(); ++i) { + buffer.append('[').append(sinks.get(i).uid).append("] = "); + } + buffer.append(op.getMnemonic()); + for (int i = 0; i < sources.size(); ++i) { + buffer.append(" [").append(sources.get(i).uid).append(']'); + } + } + else { + buffer.append(vn.toString()); + } + return buffer.toString(); + } + + /** + * Clear any incoming or outgoing edges to/from this node + */ + void clearEdges() { + sinks.clear(); + sources.clear(); + } + + @Override + public int hashCode() { + return uid; + } + + /** + * Compute a hash of the meta-data associated with the node, not including edges. + * This is effectively a 0-gram of the node, collecting data about the node itself but + * nothing about its neighbors. The hash for PcodeOp nodes, is just the opcode. For + * Varnodes, the hash collects info about the size and the type of Varnode (local, global, constant). + * @return the hash + */ + private int depthZeroHash() { + int encoding = 0; // Initialize + if (op != null) { + if (op.getOpcode() == PcodeOp.PTRSUB) { // Replace PTRSUBs with INT_ADDs + encoding = PcodeOp.INT_ADD; + } + else { + encoding = op.getOpcode(); + } + encoding |= 0xc0000000; // Bits indicating a PcodeOp specifically + } + else { + VarnodeAST node = vn; + //For Varnodes, the encoding needs to know whether the node is global or local and what + // size is allocated to it. + int ramCode = (graph.ramCaring ? 1 : 0) * (node.isPersistent() ? 1 : 0); + int constCode = (node.isConstant() ? 1 : 0); + int size = node.getSize(); + if (graph.sizeCollapse && size > 4) { // If sizeCollapse is on, treat sizes larger then 4 bytes + size = 4; // the same as a size of 4 + } + int sizeCode = ((size << 4) >> 4); // Make top 4 bits are clear + + encoding |= (ramCode << 29); + encoding |= (constCode << 28); + encoding |= sizeCode; + encoding |= (1 << 31); + if (graph.constCaring && graph.isConstantNonPointer(node)) { + // Only hash an exact constant value if it doesn't look like a pointer + return Pinning.hashTwo(encoding, (int) node.getOffset()); + } + } + return Pinning.hashTwo(encoding, 0); // Hash the collected info + } + + /** + * Compute and store a new n-gram by combining existing (n-1)-grams from sources. + * @param index is the index of the current, already computed, (n-1)-gram to recurse on + */ + void nextNGramSource(int index) { + int nextSize = 1; + DataNGram zeroGram = ngrams.get(0); // 0-gram for this node + int finalHash; + if (isCommutative()) { // Commutative nodes have indistinguishable sources. + finalHash = 0; + for (DataVertex neighbor : sources) { + DataNGram gram = neighbor.ngrams.get(index); // Immediate neighbor (n-1)-gram + finalHash += gram.hash; // Combine hashes using a commutative operation + nextSize += gram.weight; // Running tally of number of nodes in hash + } + finalHash = Pinning.hashTwo(zeroGram.hash, finalHash); + } + else { + finalHash = zeroGram.hash; + for (DataVertex neighbor : sources) { + DataNGram gram = neighbor.ngrams.get(index); // Immedate neighbor (n-1)-gram + finalHash = Pinning.hashTwo(finalHash, gram.hash); // Hash in, in order + nextSize += gram.weight; // Running tally of number of nodes in hash + } + } + + ngrams.add(new DataNGram(this, nextSize, ngrams.size(), finalHash)); + } + + /** + * @return true if this node represents a PcodeOp (as opposed to a Varnode) in the data-flow + */ + public boolean isOp() { + return (op != null); + } + + /** + * Is the underlying node a PcodeOp which takes commutative inputs? + * @return true if the PcodeOp is commutative + */ + public boolean isCommutative() { + if (op == null) { + return false; + } + int opc = op.getOpcode(); + if (opc == PcodeOp.MULTIEQUAL) { + return true; // For purposes of Pinning algorithm, treat MULTIEQUAL as commutative + } + return PcodeOp.isCommutative(opc); + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/Pinning.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/Pinning.java new file mode 100755 index 0000000000..4344f648b6 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/Pinning.java @@ -0,0 +1,911 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +import java.io.IOException; +import java.io.Writer; +import java.util.*; +import java.util.Map.Entry; + +import generic.hash.SimpleCRC32; +import generic.stl.Pair; +import ghidra.app.decompiler.*; +import ghidra.app.decompiler.component.DecompilerUtils; +import ghidra.program.model.pcode.*; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +public class Pinning { + + private static final int NGRAM_DEPTH = 24; // The (default) n-gram depth + + private DataGraph graphLeft; // Data-flow graph of the LEFT side function + private DataGraph graphRight; // Data-flow graph of the RIGHT side function + private CtrlGraph cgraphLeft; // Control-flow graph of the LEFT side function + private CtrlGraph cgraphRight; // Control-flow graph of the RIGHT side function + private Map pinMap; // Map from LEFT side (data-flow) vertices to RIGHT side vertices + private ArrayList fragments; // n-grams of data-flow vertices, in preparation for matching + private int pass; // Current pass being evaluated + private Comparator compareHashes; // Comparator for CtrlNGram sort + private Comparator compareWithinBlock; // Comparator for sorting within a basic block + + /** + * Labels for the two functions being compared by the Pinning algorithm + */ + public static enum Side { + LEFT(0), RIGHT(1); + + private int value; // Value for uid encodings + + private Side(int val) { + value = val; + } + + /** + * @return the integer encoding of the side + */ + public int getValue() { + return value; + } + } + + /** + * A data-flow vertex linked with an underlying control-flow n-gram. + */ + public static class DataCtrl { + DataVertex dataVertex; // The data-flow vertex + CtrlNGram ctrlNGram; // An n-gram of an underlying control-flow vertex + + public DataCtrl(DataVertex data, CtrlNGram ctrl) { + dataVertex = data; + ctrlNGram = ctrl; + } + + /** + * Sort by control-flow n-gram, then by side. + * Within this group, sort by block (multiple blocks can have the same hash), + * then by PcodeOp order within the block. + */ + public static class CompareWithinBlock implements Comparator { + @Override + public int compare(DataCtrl o0, DataCtrl o1) { + int hash0 = o0.ctrlNGram.hash; + int hash1 = o1.ctrlNGram.hash; + if (hash0 < hash1) { + return -1; + } + if (hash0 > hash1) { + return 1; + } + CtrlVertex o0Block = o0.ctrlNGram.root; + CtrlVertex o1Block = o1.ctrlNGram.root; + int res = o0Block.graph.side.compareTo(o1Block.graph.side); + if (res != 0) { + return res; + } + if (o0Block.uid < o1Block.uid) { + return -1; + } + if (o0Block.uid > o1Block.uid) { + return 1; + } + PcodeOp op0 = o0.dataVertex.isOp() ? o0.dataVertex.op : o0.dataVertex.vn.getDef(); + PcodeOp op1 = o1.dataVertex.isOp() ? o1.dataVertex.op : o1.dataVertex.vn.getDef(); + int order0 = op0.getSeqnum().getOrder(); + int order1 = op1.getSeqnum().getOrder(); + if (order0 < order1) { + return -1; + } + if (order0 > order1) { + return 1; + } + return 0; + } + + } + + /** + * Sort by control-flow hash, then by control-flow uid. + * Higher weight, then higher depth, hashes come first. + */ + public static class CompareHashes implements Comparator { + + @Override + public int compare(DataCtrl o1, DataCtrl o2) { + CtrlNGram o1gram = o1.ctrlNGram; + CtrlNGram o2gram = o2.ctrlNGram; + if (o1gram.weight != o2gram.weight) { + return (o1gram.weight < o2gram.weight) ? 1 : -1; // Bigger weight first + } + if (o1gram.depth != o2gram.depth) { + return (o1gram.depth < o2gram.depth) ? 1 : -1; // Bigger depth first + } + if (o1gram.hash != o2gram.hash) { + return (o1gram.hash < o2gram.hash) ? -1 : 1; + } + int res = o1gram.root.graph.side.compareTo(o2gram.root.graph.side); + if (res != 0) { + return res; + } + if (o1gram.root.uid != o2gram.root.uid) { + return (o1gram.root.uid < o2gram.root.uid) ? -1 : 1; + } + return 0; + } + + } + } + + /** + * Construct a pinning between two HighFunction + * @param hfuncLeft is the (LEFT side) HighFunction + * @param hfuncRight is the (RIGHT side) HighFunction + * @param ngramDepth is the number of n-grams to generate per node + * @param constCaring is true if the pinning should take into account exact constant values + * @param ramCaring is true if the pinning should distinguish between local and global variables + * @param castCollapse is true if CAST operations should be ignored in the pinning + * @param sizeCollapse is true if variable sizes larger than 4 should be treated as size 4 + * @param breakSym is true if symmetries should be paired arbitrarily + * @param monitor is the TaskMonitor + * @throws CancelledException if the user cancels the task + */ + public Pinning(HighFunction hfuncLeft, HighFunction hfuncRight, int ngramDepth, + boolean constCaring, boolean ramCaring, boolean castCollapse, boolean sizeCollapse, + boolean breakSym, TaskMonitor monitor) throws CancelledException { + + compareHashes = new DataCtrl.CompareHashes(); + compareWithinBlock = new DataCtrl.CompareWithinBlock(); + pinMap = new HashMap<>(); + + graphLeft = + new DataGraph(Side.LEFT, hfuncLeft, constCaring, ramCaring, castCollapse, sizeCollapse); + graphRight = new DataGraph(Side.RIGHT, hfuncRight, constCaring, ramCaring, castCollapse, + sizeCollapse); + // Control-flow graphs are used to break ties when matching data-flow + cgraphLeft = new CtrlGraph(Side.LEFT, hfuncLeft); + cgraphRight = new CtrlGraph(Side.RIGHT, hfuncRight); + + // Compute n-gram hashes + graphLeft.makeNGrams(ngramDepth); + graphRight.makeNGrams(ngramDepth); + cgraphLeft.makeNGrams(ngramDepth); + cgraphRight.makeNGrams(ngramDepth); + + makeFragments(); // Put data-flow hashes in sorted list for matching + doPinning(ngramDepth, breakSym, monitor); + } + + /** + * Update control-flow n-grams with matching info from previous passes, to help + * distinguish between similar nodes. + * @param ngramDepth is the maximum depth of n-gram being to generate + */ + private void updateCtrlHashes(int ngramDepth) { + + cgraphLeft.clearNGrams(); + cgraphRight.clearNGrams(); + HashSet seen = new HashSet<>(); + // Recalculate (some) control-flow 0-grams, based on matching data-flow in them + for (DataVertex node : pinMap.keySet()) { + if (!node.isOp()) { + continue; + } + CtrlVertex cnode = dataToCtrl(node); + if (!seen.contains(cnode)) { + CtrlVertex csidekick = dataToCtrl(pinMap.get(node)); + seen.add(cnode); + int flavor = cnode.uid; // Flavor to add to the 0-grams + cnode.setZeroGram(flavor); // Recompute 0-gram + csidekick.setZeroGram(flavor); // with matching flavor + } + } + + cgraphLeft.makeNGrams(ngramDepth); // Recompute all n-grams, recursively including new 0-grams + cgraphRight.makeNGrams(ngramDepth); + } + + /** + * Go back through data-flow vertices that were collapsed out of the original graphs. + * Each collapsed DataVertex is associated to an uncollapsed vertex. If an uncollapsed vertex + * has a match in the other graph also with associated collapsed vertices, we attempt + * to pair the two sets of collapsed vertices, just based on PcodeOp opcodes. + */ + private void pinAssociates() { + DataGraph.Associate associate1 = new DataGraph.Associate(null, 0); + for (Entry> entry : graphLeft.associates + .entrySet()) { + ArrayList side0 = entry.getValue(); + associate1.node = pinMap.get(entry.getKey().node); + if (associate1.node == null) { + continue; + } + associate1.slot = entry.getKey().slot; + ArrayList side1 = graphRight.associates.get(associate1); + if (side1 == null) { + continue; + } + if (side0.size() != side1.size() || side0.size() > 4) { + continue; + } + boolean matching = true; + for (int i = 0; i < side0.size(); i += 2) { + DataVertex op0 = side0.get(i); + DataVertex op1 = side1.get(i); + if (op0.op.getOpcode() != op1.op.getOpcode()) { + matching = false; + break; + } + } + if (matching) { + for (int i = 0; i < side0.size(); ++i) { + DataVertex v0 = side0.get(i); + DataVertex v1 = side1.get(i); + if (v0.paired || v1.paired) { + continue; + } + establishMatch(v0, v1); + } + } + } + } + + /** + * Creates the sorted list of n-gram hashes used to decide which nodes will get pinned. + */ + private void makeFragments() { + fragments = new ArrayList<>(); + for (int side = 0; side < 2; side++) { // Collect n-grams from both sides + DataGraph graph = (side == 0 ? graphLeft : graphRight); + for (DataVertex node : graph.nodeList) { // for all data-flow vertices + for (int d = 0; d < node.ngrams.size(); d++) { // and for every possible depth + fragments.add(node.ngrams.get(d)); // into one list + } + } + } + + // Sort the list by weight and hash so that possible matches are adjacent + Collections.sort(fragments); + } + + /** + * Given a list of more than 2 DataNGrams with matching hash, try to distinguish the underlying + * DataVertex objects by comparing CtrlNGrams associated with each DataVertex through its + * containing CtrlVertex. If DataVertex pairs can be distinguished, they are added to pinMap. + * @param matchList is the list of matching DataNGrams + * @param useOrder is true if block order should be used to break ties + * @param monitor is the TaskMonitor + * @throws CancelledException if the user cancels the task + */ + private void breakTieWithCtrlFlow(ArrayList matchList, boolean useOrder, + TaskMonitor monitor) throws CancelledException { + DataNGram current = matchList.get(0); + if (current.weight <= 1) { + return; // Don't try to break ties on low weight n-grams + } + ArrayList cfragsList = new ArrayList<>(); + + // Create the list of control-flow n-grams, and set up a way to get back to the original DataVertex + for (int j = 0; j < matchList.size(); j++) { + monitor.checkCancelled(); + DataVertex tiedVertex = matchList.get(j).root; + + // Mark the vertex as having been analyzed this pass. This prevents additional + // rounds of tie breaking on the same set of nodes for lower depth n-grams. I.e., + // if vertices are indistinguishable at this depth, they will continue to be + // indistinguishable at lower depths. + tiedVertex.passComplete = pass; + + // The vertex is guaranteed to have at least 1 source, as the weight > 1 + DataVertex opVertex = (current.root.isOp() ? tiedVertex : tiedVertex.sources.get(0)); + CtrlVertex ctiedVertex = dataToCtrl(opVertex); + for (int d = 0; d < ctiedVertex.ngrams.size(); d++) { + CtrlNGram nGram = ctiedVertex.ngrams.get(d); + DataCtrl temp = new DataCtrl(tiedVertex, nGram); // Tie CtrlNGram to tiedVertex + cfragsList.add(temp); + } + } + + cfragsList.sort(compareHashes); // Sort list so that identical n-grams are adjacent + int j = 0; + while (j < cfragsList.size()) { + if (monitor.isCancelled()) { + return; + } + DataCtrl ccurrent = cfragsList.get(j); + if (ccurrent.dataVertex.paired) { // If we've already paired DataVertex + j++; + continue; // Don't look at it again + } + + int jbar = j + 1; + while (jbar < cfragsList.size()) { // Iterate to the first n-gram whose hash doesn't match + DataCtrl cfuture = cfragsList.get(jbar); + if (!ccurrent.ctrlNGram.equalHash(cfuture.ctrlNGram)) { + break; + } + jbar++; + } + + if (jbar - j == 2) { // Exactly 2 matching n-gram hashes. Possible pair. + DataCtrl cnext = cfragsList.get(j + 1); + if (ccurrent.ctrlNGram.graphsDiffer(cnext.ctrlNGram)) { + DataVertex temp0 = ccurrent.dataVertex; + DataVertex temp1 = cnext.dataVertex; + ngramPinner(temp0, temp1, current.depth); + } + } + else if (useOrder && jbar - j > 2) { + breakTieUsingOrder(cfragsList, j, jbar, current); + } + j = jbar; + + } + } + + /** + * Test if a range of vertices that have a matching data n-gram and a matching control-flow n-gram + * occur within a single pair of basic blocks. There must be an equal number of vertices + * in one basic block on the LEFT and in one on the RIGHT. Additionally, the vertices must + * either have no output edges or be MULTIEQUAL op nodes. + * @param frags is the list of vertices associated with control-flow n-grams + * @param start is the start of the matching range + * @param stop is the end of the matching range + * @return true if the vertices occur within a single pair of basic blocks and can be paired + */ + private static boolean isBlockPair(ArrayList frags, int start, int stop) { + int leftCount = 0; + int rightCount = 0; + int leftUid = -1; + int rightUid = -1; + for (int i = start; i < stop; ++i) { + DataCtrl frag = frags.get(i); + DataVertex vert = frag.dataVertex; + if (!vert.sinks.isEmpty() && + (!vert.isOp() || vert.op.getOpcode() != PcodeOp.MULTIEQUAL)) { + // Nodes must be terminal roots of data-flow or MULTIEQUAL ops + return false; + } + CtrlVertex cvert = frag.ctrlNGram.root; + if (cvert.graph.side == Side.LEFT) { + leftCount += 1; + if (leftCount == 1) { + leftUid = cvert.uid; + } + else if (leftUid != cvert.uid) { + return false; // More than one block on LEFT side + } + } + else { + rightCount += 1; + if (rightCount == 1) { + rightUid = cvert.uid; + } + else if (rightUid != cvert.uid) { + return false; // More than one block on RIGHT side + } + } + } + return (leftCount == rightCount); + } + + /** + * Given a range of vertices with identical data n-grams and associated control-flow n-grams, + * test if they all occur in 1 basic block (pair). If they can, match the vertices + * in the order they occur within the basic block. The new paired vertices are added to the pinMap. + * @param frags is the list of vertices with associated control-flow n-grams + * @param start is the start of the matching range + * @param stop is the end of the matching range + * @param firstNGram is the matching data n-gram + */ + private void breakTieUsingOrder(ArrayList frags, int start, int stop, + DataNGram firstNGram) { + + if (isBlockPair(frags, start, stop)) { + List subList = frags.subList(start, stop); + subList.sort(compareWithinBlock); + int size = (stop - start) / 2; + for (int i = 0; i < size; ++i) { + ngramPinner(subList.get(i).dataVertex, subList.get(i + size).dataVertex, + firstNGram.depth); + } + } + } + + /** + * Starting at the given index in fragment, the main n-gram list, collect all n-grams + * with the same hash. The matching n-grams are passed back in matchList. + * Any n-gram whose weight is less than the minWeight threshold is skipped, as + * is any n-gram whose underlying DataVertex is already paired or ruled out. + * @param i is the given starting index + * @param matchList will contain exactly the list of matching n-grams being passed back + * @param minWeight is the minimum weight threshold for n-grams + * @return the index advanced to the next unexamined slot + */ + private int collectEqualHash(int i, ArrayList matchList, int minWeight) { + DataNGram first = null; + matchList.clear(); + for (;;) { + if (i >= fragments.size()) { + return i; + } + first = fragments.get(i); + i += 1; + if (first.weight < minWeight) { + if (!first.root.isOp() || first.root.op.getOpcode() != PcodeOp.CALL) { + continue; + } + } + if (!first.root.paired && first.root.passComplete < pass) { + break; + } + } + matchList.add(first); + for (;;) { + if (i >= fragments.size()) { + return i; + } + DataNGram gram = fragments.get(i); + if (!first.equalHash(gram)) { + break; + } + i += 1; + if (!gram.root.paired && gram.root.passComplete < pass) { + matchList.add(gram); + } + } + return i; + } + + /** + * Pair the two given data-flow vertices. + * @param left is the LEFT side vertex + * @param right is the RIGHT side vertex + */ + private void establishMatch(DataVertex left, DataVertex right) { + pinMap.put(left, right); + left.paired = true; + right.paired = true; + } + + /** + * Do one pass of the pinning algorithm. The n-gram list is run through once. For each set of + * n-grams with matching hashes, if there are exactly 2, the associated data-flow vertices are paired. + * If there are more then 2, we attempt to "break the tie" by associating control-flow n-grams to + * the subset of data-flow vertices and pairing these. + * @param minWeight is the weight threshold for considering an n-gram in the list for matching + * @param useOrder is true if block and operand order should be used to break ties + * @param monitor is the TaskMonitor + * @throws CancelledException if the user cancels the task + */ + private void pinMain(int minWeight, boolean useOrder, TaskMonitor monitor) + throws CancelledException { + int i = 0; // The main index into fragments, the n-gram list + monitor.setMessage("Pinning all..."); + monitor.setIndeterminate(false); + monitor.initialize(fragments.size()); + ArrayList matchList = new ArrayList<>(); + while (i < fragments.size()) { + if (i % 1000 == 0) { + monitor.setProgress(i); + } + i = collectEqualHash(i, matchList, minWeight); + + if (matchList.size() == 2) { // Exactly 2 matching n-grams. Possible pair. + DataNGram gram0 = matchList.get(0); + DataNGram gram1 = matchList.get(1); + if (gram0.graphsDiffer(gram1)) { // Check that one n-gram comes from each side + DataVertex left = gram0.root.graph.side == Side.LEFT ? gram0.root : gram1.root; + DataVertex right = + gram0.root.graph.side == Side.RIGHT ? gram0.root : gram1.root; + ngramPinner(left, right, gram0.depth); // Pin the match and everything above it + } + } + else if (matchList.size() > 2) { // More then 2 matching n-grams + breakTieWithCtrlFlow(matchList, useOrder, monitor); + if (useOrder) { + matchViaOperator(matchList); + } + } + } + } + + /** + * Run the full pinning algorithm. Continue doing passes over the n-gram list until no + * further pairs are found. + * @param nGramDepth is the maximum depth of n-gram to use + * @param breakSym is true if a symmetry breaking pass should be performed at the end + * @param monitor is the TaskMonitor + * @throws CancelledException if the user cancels the task + */ + private void doPinning(int nGramDepth, boolean breakSym, TaskMonitor monitor) + throws CancelledException { + pass = 0; + pinMain(2, false, monitor); // First pass, using weight threshold of 2 + cgraphLeft.addEdgeColor(); // Try to distinguish control-flow nodes further + cgraphRight.addEdgeColor(); + boolean checkForTies = true; + while (checkForTies) { // Continue until no further pairs are found + pass += 1; + int lastPinSize = pinMap.size(); + updateCtrlHashes(nGramDepth); + pinMain(0, false, monitor); + checkForTies = (lastPinSize != pinMap.size()); + } + if (breakSym) { + checkForTies = true; + while (checkForTies) { + pass += 1; + int lastPinSize = pinMap.size(); + pinMain(0, true, monitor); // Use block and operand order to break ties + checkForTies = (lastPinSize != pinMap.size()); + } + } + pinAssociates(); // Try to pair nodes in the original graph that were collapsed out + } + + /** + * Pin the given pair of data-flow vertices, then recursively try to pin vertices + * by following the immediate incoming data-flow edge. The pair is made for a specific n-gram + * depth. Recursive pairing stops at any point where n-grams don't match up to this depth. + * Edges into commutative PcodeOp nodes are also disambiguated with n-gram hashes up to this depth. + * @param left is the LEFT side data-flow vertex to pair + * @param right is the RIGHT side data-flow vertex to pair + * @param depth is the specific n-gram depth for the pair + */ + private void ngramPinner(DataVertex left, DataVertex right, int depth) { + + if (depth < 0 || left == null || left.paired || right.paired) { + return; + } + + establishMatch(left, right); // Formally pair the vertices + + boolean goForIt; + + if (left.isCommutative()) { // Nodes whose incoming edges must be disambiguated + if (left.op.getOpcode() != PcodeOp.MULTIEQUAL) { + DataVertex left0 = left.sources.get(0); + DataVertex left1 = left.sources.get(1); + DataVertex right0 = right.sources.get(0); + DataVertex right1 = right.sources.get(1); + int lasty = left.ngrams.size() - 1; + + if (left0.ngrams.get(lasty).hash == left1.ngrams.get(lasty).hash || + right0.ngrams.get(lasty).hash == right1.ngrams.get(lasty).hash) { + return; + } + + } + for (DataVertex srcLeft : left.sources) { // For each incoming edge of the LEFT side vertex + if (srcLeft.paired) { + continue; + } + for (DataVertex srcRight : right.sources) { // Search for a matching edge on RIGHT side + if (srcRight.paired) { + continue; + } + goForIt = true; + for (int i = 0; i < Math.max(1, depth); i++) { // n-grams must match up to given depth + if (srcLeft.ngrams.get(i).hash != srcRight.ngrams.get(i).hash) { + goForIt = false; + break; + } + } + if (goForIt) { // If all hashes match for the two edges, pair them + // Try to extend depth of matching hashes before pairing the edges + int newDepth = Math.max(depth - 1, 0); + while (-1 < newDepth && newDepth < srcLeft.ngrams.size() && srcLeft.ngrams + .get(newDepth).hash == srcRight.ngrams.get(newDepth).hash) { + newDepth++; + } + ngramPinner(srcLeft, srcRight, newDepth - 1); // Recursively match these edges + break; + } + } + } + } + else { // Edges are paired in order + if (left.sources.size() == right.sources.size()) { + for (int n = 0; n < left.sources.size(); n++) { + + DataVertex srcLeft = left.sources.get(n); + DataVertex srcRight = right.sources.get(n); + goForIt = true; + for (int i = 0; i < Math.max(1, depth); i++) { // n-grams must match up to given depth + if (srcLeft.ngrams.get(i).hash != srcRight.ngrams.get(i).hash) { + goForIt = false; + break; + } + } + if (goForIt) { // If all hashes match for the two edges, pair them + // Try to extend depth of matching hashes before pairing edges + int i = Math.max(depth - 1, 0); + while (-1 < i && i < srcLeft.ngrams.size() && + srcLeft.ngrams.get(i).hash == srcRight.ngrams.get(i).hash) { + i++; + } + ngramPinner(srcLeft, srcRight, i - 1); // Recursively match these edges + } + } + } + } + } + + /** + * Given a data-flow representing a PcodeOp, return the control-flow vertex containing the PcodeOp. + * @param node is the data-flow vertex + * @return the containing control-flow vertex + */ + private CtrlVertex dataToCtrl(DataVertex node) { + PcodeBlockBasic parent = node.op.getParent(); + CtrlGraph whichGraph = (node.graph == graphLeft ? cgraphLeft : cgraphRight); + return whichGraph.blockToVertex.get(parent); + } + + /** + * Find the RIGHT side Varnode matching the given LEFT side Varnode. + * The LEFT and RIGHT sides are determined by the order HighFunctions are given to the constructor. + * @param vnLeft is the given Varnode from LEFT side + * @return the matching Varnode from RIGHT side or null + */ + public VarnodeAST findMatch(Varnode vnLeft) { + DataVertex vertLeft = graphLeft.vnToVert.get(vnLeft); + if (vertLeft == null) { + return null; + } + DataVertex vertRight = pinMap.get(vertLeft); + if (vertRight != null) { + return vertRight.vn; + } + return null; + } + + /** + * Find the RIGHT side PcodeOp matching the given LEFT side PcodeOp. + * The LEFT and RIGHT sides are determined by the order HighFunctions are given to the constructor. + * @param opLeft is the given PcodeOp from LEFT side + * @return the matching PcodeOp from RIGHT side or null + */ + public PcodeOpAST findMatch(PcodeOp opLeft) { + DataVertex vertLeft = graphLeft.opToVert.get(opLeft); + if (vertLeft == null) { + return null; + } + DataVertex vertRight = pinMap.get(vertLeft); + if (vertRight != null) { + return vertRight.op; + } + return null; + } + + /** + * Determine if a token should be be filtered from match display. + * Some tokens like "," and " " may be attached to matchable operations but can + * clutter the display if they are highlighted for a match. + * @param token is the specific token to check + * @return true if the token should not be highlighted as part of a match + */ + private boolean filterToken(ClangToken token) { + String text = token.getText(); + if (text.length() == 0) { + return true; + } + if (text.length() == 1) { + char c = text.charAt(0); + if (c == ' ' || c == ',') { + return true; + } + } + + if (token instanceof ClangTypeToken) { + return true; + } + return false; + } + + /** + * Build a TokenBin for every DataVertex on both sides. Pair a TokenBin with its match, + * if its underlying DataVertex is paired. + * @param leftTokenGp are the tokens for LEFT side + * @param rightTokenGp are the tokens for RIGHT side + * @return a single list of LEFT side TokenBins then RIGHT side TokenBins + */ + public ArrayList buildTokenMap(ClangTokenGroup leftTokenGp, + ClangTokenGroup rightTokenGp) { + + HashMap, TokenBin> lvertToBin = new HashMap<>(); + HashMap, TokenBin> rvertToBin = new HashMap<>(); + + for (int side = 0; side < 2; side++) { + // side == 0: set up the left side + // side == 1: set up the right side + ClangTokenGroup tokGp = (side == 0 ? leftTokenGp : rightTokenGp); + DataGraph graph = (side == 0 ? graphLeft : graphRight); + HashMap, TokenBin> vertToBin = + (side == 0 ? lvertToBin : rvertToBin); + + ArrayList nodes = new ArrayList<>(); + tokGp.flatten(nodes); + for (ClangNode node : nodes) { + if (node instanceof ClangToken tok) { + if (filterToken(tok)) { + continue; + } + VarnodeAST vn = (VarnodeAST) DecompilerUtils.getVarnodeRef(tok); + PcodeOpAST op = (PcodeOpAST) tok.getPcodeOp(); + + DataVertex opNode = graph.opToVert.get(op); + DataVertex vnNode = graph.vnToVert.get(vn); + Pair nodePair = new Pair<>(opNode, vnNode); + + if (!vertToBin.containsKey(nodePair)) { + vertToBin.put(nodePair, new TokenBin(graph.getHighFunction())); + } + vertToBin.get(nodePair).add(tok); + } + } + } + + // Match a TokenBin if its underlying DataVertex is paired + ArrayList highBins = new ArrayList<>(); + for (Pair lNodePair : lvertToBin.keySet()) { + TokenBin lbin = lvertToBin.get(lNodePair); + DataVertex lkey = lNodePair.first; + DataVertex lval = lNodePair.second; + DataVertex rkey = null; + DataVertex rval = null; + if (lkey != null && lkey.paired) { + rkey = pinMap.get(lkey); + } + if (lval != null && lval.paired) { + rval = pinMap.get(lval); + } + + if (((lkey == null) != (rkey == null)) || ((lval == null) != (rval == null))) { + continue; + } + if ((rkey == null && rval == null) || (lkey == null && lval == null)) { + continue; + } + + Pair rNodePair = new Pair<>(rkey, rval); + if (rvertToBin.containsKey(rNodePair)) { + TokenBin rbin = rvertToBin.get(rNodePair); + lbin.sidekick = rbin; + rbin.sidekick = lbin; + } + } + + // Put everything into the final list + lvertToBin.remove(new Pair(null, null)); + rvertToBin.remove(new Pair(null, null)); + highBins.addAll(lvertToBin.values()); + highBins.addAll(rvertToBin.values()); + + return highBins; + } + + /** + * Dump a string representation of pinning data structures (for debugging). + * @param writer is the stream to write the string to. + * @throws IOException for problems writing to the stream + */ + public void dump(Writer writer) throws IOException { + graphLeft.dump(writer); + graphRight.dump(writer); +// for (DataNGram vertex : fragments) { +// writer.append(vertex.toString()); +// writer.append("\n"); +// } + for (DataVertex vertex : graphLeft.nodeList) { + DataVertex match = pinMap.get(vertex); + if (match != null) { + writer.append("match "); + writer.append(Integer.toString(vertex.uid)); + writer.append(" to "); + writer.append(Integer.toString(match.uid)); + writer.append("\n"); + } + } + } + + /** + * Match data-flow between two HighFunctions. The matching algorithm is performed immediately. + * The resulting Pinning object can be queried for matches via + * - findMatch(Varnode) or + * - findMatch(PcodeOp) + * + * ClangToken matches can be calculated by calling buildTokenMap(). + * @param hfuncLeft is the LEFT side function + * @param hfuncRight is the RIGHT side function + * @param matchConstantsExactly is true if (small) constant values should be forced to match + * @param sizeCollapse is true if variable sizes larger than 4 should be treated as size 4 + * @param breakSym is true if code symmetries should be ordered arbitrarily so they can be paired + * @param monitor is the TaskMonitor + * @return the populated Pinning object + * @throws CancelledException if the user cancels the task + */ + public static Pinning makePinning(HighFunction hfuncLeft, HighFunction hfuncRight, + boolean matchConstantsExactly, boolean sizeCollapse, boolean breakSym, + TaskMonitor monitor) throws CancelledException { + + boolean matchRamSpace = true; + boolean castCollapse = true; + // Make the pinning, the map from the data graph of hfuncLeft to that of hfuncRight. + Pinning pin = new Pinning(hfuncLeft, hfuncRight, NGRAM_DEPTH, matchConstantsExactly, + matchRamSpace, castCollapse, sizeCollapse, breakSym, monitor); + + return pin; + } + + /** + * Try to pair vertices that are inputs to the same commutative operator. + * Reaching here, the vertices could not be distinguished by any other method, so + * we order the vertices based on their input slot to the operator. + * @param ngrams is the set of matching n-grams + */ + private void matchViaOperator(ArrayList ngrams) { + DataNGram firstNGram = ngrams.get(0); + for (DataNGram ngram : ngrams) { + DataVertex vertLeft = ngram.root; + if (vertLeft.graph.side != Side.LEFT) { + continue; + } + for (int j = 0; j < vertLeft.sinks.size(); ++j) { + DataVertex opVertLeft = vertLeft.sinks.get(j); + if (!opVertLeft.isOp() || !opVertLeft.isCommutative() || !opVertLeft.paired) { + continue; + } + DataVertex opVertRight = pinMap.get(opVertLeft); + if (opVertLeft.sources.size() != opVertRight.sources.size()) { + continue; + } + for (int i = 0; i < opVertLeft.sources.size(); ++i) { + DataVertex tVertLeft = opVertLeft.sources.get(i); + DataVertex tVertRight = opVertRight.sources.get(i); + if (tVertLeft.paired || tVertRight.paired) { + continue; + } + int index = tVertLeft.ngrams.size() - 1; + if (!tVertLeft.ngrams.get(index).equalHash(firstNGram)) { + continue; + } + if (!tVertRight.ngrams.get(index).equalHash(firstNGram)) { + continue; + } + ngramPinner(tVertLeft, tVertRight, firstNGram.depth); + } + } + } + } + + /** + * The function we use to hash two integers into one. Used in multiple places in the pinning algorithm. + * @param first is the first integer to hash + * @param second is the second integer + * @return the hash of the two integers + */ + static int hashTwo(int first, int second) { + int result = 0; + for (int i = 0; i < 4; i++) { + result = SimpleCRC32.hashOneByte(result, first >> i * 8); + } + for (int i = 0; i < 4; i++) { + result = SimpleCRC32.hashOneByte(result, second >> i * 8); + } + return result; + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/TokenBin.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/TokenBin.java new file mode 100755 index 0000000000..5e60738f97 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/codecompare/graphanalysis/TokenBin.java @@ -0,0 +1,110 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.codecompare.graphanalysis; + +import java.util.*; + +import ghidra.app.decompiler.ClangToken; +import ghidra.program.model.pcode.HighFunction; +import ghidra.util.Msg; + +/** + * An iterable list of Decompiler tokens {@link ClangToken} in one window that is paired with a + * list in another window. The matching TokenBin, if it exists, is obtained by calling + * getMatch(). This container is constructed and populated only by {@link Pinning}. + */ +public class TokenBin implements Iterable { + private ArrayList bin; // The list of tokens in this bin + private HighFunction highFunction; // The function owning the tokens + TokenBin sidekick; // The bin (from the other window) matching this + + /** + * Construct an (initially empty) token bin. + * @param highFunction is the function the bin is associated with + */ + TokenBin(HighFunction highFunction) { + this.bin = new ArrayList<>(); + this.highFunction = highFunction; + this.sidekick = null; + } + + /** + * @return the HighFunction owning the tokens in this bin + */ + public HighFunction getHighFunction() { + return highFunction; + } + + /** + * @return the TokenBin paired with this + */ + public TokenBin getMatch() { + return sidekick; + } + + /** + * Get the i-th token in this bin + * @param i is the index of the token + * @return the selected token + */ + public ClangToken get(int i) { + return bin.get(i); + } + + /** + * @return the number of tokens in this bin + */ + public int size() { + return bin.size(); + } + + @Override + public Iterator iterator() { + return bin.iterator(); + } + + /** + * Add a new token to this bin. + * @param newToken is the token to add + */ + void add(ClangToken newToken) { + // Check to make sure we're in the right function. + HighFunction tokenHighFunction = newToken.getClangFunction().getHighFunction(); + if (!highFunction.equals(tokenHighFunction)) { + Msg.warn(this, + "ERROR: Trying to ADD token '" + newToken.getText() + "' to incorrect TokenBin."); + return; + } + bin.add(newToken); // Add token to the list + } + + /** + * From a list of TokenBins, find the (first) one containing a particular ClangToken + * @param highBins is the list of bins + * @param myToken is the ClangToken to find + * @return the first bin containing the token, or null + */ + public static TokenBin getBinContainingToken(List highBins, ClangToken myToken) { + for (TokenBin bin : highBins) { + for (ClangToken token : bin.bin) { + if (myToken == token) { + return bin; + } + } + } + return null; + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/CodeCompareAddressCorrelation.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/CodeCompareAddressCorrelation.java new file mode 100755 index 0000000000..0cbf8cdeea --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/CodeCompareAddressCorrelation.java @@ -0,0 +1,455 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api.correlator.address; + +import java.util.*; +import java.util.Map.Entry; + +import ghidra.app.decompiler.*; +import ghidra.codecompare.graphanalysis.Pinning; +import ghidra.codecompare.graphanalysis.TokenBin; +import ghidra.program.model.address.*; +import ghidra.program.model.block.*; +import ghidra.program.model.listing.*; +import ghidra.program.model.pcode.HighFunction; +import ghidra.program.util.*; +import ghidra.util.Msg; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +public class CodeCompareAddressCorrelation implements AddressCorrelation { + static enum CorrelationKind { + CODE_COMPARE, LCS, PARAMETERS; + } + + static class CorrelationContainer { + public static boolean USE_RANDOM_CC_COLORS = false; + public final CorrelationKind kind; + public final AddressRange range; + + public CorrelationContainer(CorrelationKind kind, AddressRange range) { + this.kind = kind; + this.range = range; + } + } + + private static final int TIMEOUT_SECONDS = 60; + + private final Function sourceFunction; + private final Function destinationFunction; + private final Program sourceProgram; + private final Program destinationProgram; + + private Map cachedForwardAddressMap; + + public CodeCompareAddressCorrelation(Function sourceFunction, Function destinationFunction) { + this.sourceFunction = sourceFunction; + this.destinationFunction = destinationFunction; + this.sourceProgram = sourceFunction.getProgram(); + this.destinationProgram = destinationFunction.getProgram(); + + DebugUtils.enable(false); // Set to "true" to enable debugging or "false" to disable. + } + + @Override + public AddressRange getCorrelatedDestinationRange(Address sourceAddress, TaskMonitor monitor) + throws CancelledException { + initialize(monitor); + + CorrelationContainer container = cachedForwardAddressMap.get(sourceAddress); + if (container == null) { + return null; + } + return container.range; + } + + private static final Comparator CUCOMPARATOR = new Comparator() { + @Override + public int compare(CodeUnit o1, CodeUnit o2) { + return o1.getAddress().compareTo(o2.getAddress()); + } + }; + + private void initialize(TaskMonitor monitor) throws CancelledException { + if (cachedForwardAddressMap == null) { + cachedForwardAddressMap = new HashMap(); + + HashMap sourceToDestinationPairings = new HashMap(); + HashMap destinationToSourcePairings = new HashMap(); + + AddressSet sourceSet = new AddressSet(); + AddressSet destinationSet = new AddressSet(); + + TreeMap> sourceMap = + new TreeMap>(CUCOMPARATOR); + TreeMap> destinationMap = + new TreeMap>(CUCOMPARATOR); + + processCodeCompare(monitor, sourceToDestinationPairings, destinationToSourcePairings, + sourceSet, destinationSet, sourceMap, destinationMap); + + // now, if we're on the same architecture, try to LCS the + // interstices between our golden matches + + if (sourceFunction.getProgram() + .getLanguage() + .getProcessor() + .equals(destinationFunction.getProgram().getLanguage().getProcessor())) { + processLCSBlocks(monitor, sourceToDestinationPairings, sourceSet, destinationSet, + sourceMap, destinationMap); + } + + DebugUtils.processMap(sourceMap, sourceProgram); + DebugUtils.processMap(destinationMap, destinationProgram); + + DebugUtils.colorize(cachedForwardAddressMap, sourceProgram, destinationProgram); + } + } + + private void processCodeCompare(TaskMonitor monitor, + HashMap sourceToDestinationPairings, + HashMap destinationToSourcePairings, AddressSet sourceSet, + AddressSet destinationSet, Map> sourceMap, + Map> destinationMap) throws CancelledException { + // compute the mapping by means of code compare + + DecompInterface sourceDecompiler = null; + DecompInterface destinationDecompiler = null; + + try { + sourceDecompiler = new DecompInterface(); + destinationDecompiler = new DecompInterface(); + + sourceDecompiler.openProgram(sourceProgram); + destinationDecompiler.openProgram(destinationProgram); + + DecompileResults sourceResults = + sourceDecompiler.decompileFunction(sourceFunction, TIMEOUT_SECONDS, monitor); + DecompileResults destinationResults = destinationDecompiler + .decompileFunction(destinationFunction, TIMEOUT_SECONDS, monitor); + + ClangTokenGroup sourceMarkup = sourceResults.getCCodeMarkup(); + ClangTokenGroup destinationMarkup = destinationResults.getCCodeMarkup(); + + HighFunction sourceHFunc = sourceResults.getHighFunction(); + HighFunction destinationHFunc = destinationResults.getHighFunction(); + String errorMessage = ""; + if (sourceHFunc == null) { + errorMessage += (" Source Decompiler failed to get function. " + + sourceResults.getErrorMessage()); + } + if (destinationHFunc == null) { + errorMessage += (" Destination Decompiler failed to get function. " + + destinationResults.getErrorMessage()); + } + if (!errorMessage.isEmpty()) { + // For now throw a RuntimeException to see what failed in the decompiler, + // otherwise we will just get a NullPointerException. + throw new RuntimeException(errorMessage); + } + + boolean matchConstantsExactly = false; + int srcsize = sourceProgram.getLanguage().getLanguageDescription().getSize(); + int destsize = destinationProgram.getLanguage().getLanguageDescription().getSize(); + boolean sizeCollapse = (srcsize != destsize); + Pinning pin = Pinning.makePinning(sourceHFunc, destinationHFunc, matchConstantsExactly, + sizeCollapse, true, monitor); + ArrayList highBins = pin.buildTokenMap(sourceMarkup, destinationMarkup); + + for (TokenBin tokenBin : highBins) { + if (tokenBin.getMatch() == null || tokenBin.getMatch().size() != tokenBin.size()) { + continue; + } + boolean isSourceBin = tokenBin.getHighFunction().equals(sourceHFunc); + + for (int ii = 0; ii < tokenBin.size(); ++ii) { + ClangToken binToken = tokenBin.get(ii); + ClangToken sidekickToken = tokenBin.getMatch().get(ii); + + ClangToken sourceToken = isSourceBin ? binToken : sidekickToken; + ClangToken destinationToken = isSourceBin ? sidekickToken : binToken; + + Address srcStart = sourceToken.getMinAddress(); + Address srcEnd = sourceToken.getMaxAddress(); + + Address destStart = destinationToken.getMinAddress(); + Address destEnd = destinationToken.getMaxAddress(); + + if (destStart == null || destEnd == null || srcStart == null || + srcEnd == null) { + continue; + } + AddressSet sourceIntersection = sourceSet.intersectRange(srcStart, srcEnd); + AddressSet destinationIntersection = + destinationSet.intersectRange(destStart, destEnd); + + if (!sourceIntersection.isEmpty() || !destinationIntersection.isEmpty()) { + continue; + } + + DebugUtils.recordEOLComment(sourceMap, sourceProgram, srcStart, srcEnd, + destinationProgram, destStart, destEnd); + DebugUtils.recordEOLComment(destinationMap, destinationProgram, destStart, + destEnd, sourceProgram, srcStart, srcEnd); + + sourceSet.addRange(srcStart, srcEnd); + destinationSet.addRange(destStart, destEnd); + + sourceToDestinationPairings.put(srcStart, destStart); + destinationToSourcePairings.put(destStart, srcStart); + + AddressRangeImpl range = new AddressRangeImpl(destStart, destEnd); + CorrelationContainer container = + new CorrelationContainer(CorrelationKind.CODE_COMPARE, range); + + // Assign container to all addresses: srcStart to srcEnd inclusive + while (!srcStart.equals(srcEnd)) { + cachedForwardAddressMap.put(srcStart, container); + srcStart = srcStart.addNoWrap(1); + } + cachedForwardAddressMap.put(srcStart, container); + } + } + } + catch (AddressOverflowException e) { + Msg.error(this, + "Unexpected address overflow; token's range didn't span continguous region", e); + } + finally { + if (sourceDecompiler != null) { + sourceDecompiler.dispose(); + } + if (destinationDecompiler != null) { + destinationDecompiler.dispose(); + } + } + } + + private void processLCSBlocks(TaskMonitor monitor, + HashMap sourceToDestinationPairings, AddressSet sourceSet, + AddressSet destinationSet, Map> sourceMap, + Map> destinationMap) throws CancelledException { + CodeBlockModel sourceBlockModel = new BasicBlockModel(sourceProgram); + CodeBlockModel destinationBlockModel = new BasicBlockModel(destinationProgram); + + Listing sourceListing = sourceProgram.getListing(); + Listing destinationListing = destinationProgram.getListing(); + + Set> entrySet = sourceToDestinationPairings.entrySet(); + for (Entry entry : entrySet) { + Address sourceAddress = entry.getKey(); + Address destinationAddress = entry.getValue(); + + CodeBlock[] sourceBlocks = + sourceBlockModel.getCodeBlocksContaining(sourceAddress, monitor); + CodeBlock[] destinationBlocks = + destinationBlockModel.getCodeBlocksContaining(destinationAddress, monitor); + + if (sourceBlocks != null && destinationBlocks != null) { + if (sourceBlocks.length == 1 && destinationBlocks.length == 1) { + // work backwards + + CodeUnitIterator sourceCodeUnitIterator = + sourceListing.getCodeUnits(sourceAddress, false); + CodeUnitIterator destinationCodeUnitIterator = + destinationListing.getCodeUnits(destinationAddress, false); + + processLCSCodeUnits(monitor, sourceSet, destinationSet, sourceMap, + destinationMap, sourceCodeUnitIterator, destinationCodeUnitIterator); + + // now work forwards + sourceCodeUnitIterator = sourceListing.getCodeUnits(sourceAddress, true); + destinationCodeUnitIterator = + destinationListing.getCodeUnits(destinationAddress, true); + + processLCSCodeUnits(monitor, sourceSet, destinationSet, sourceMap, + destinationMap, sourceCodeUnitIterator, destinationCodeUnitIterator); + } + } + } + } + + private void processLCSCodeUnits(TaskMonitor monitor, AddressSet sourceSet, + AddressSet destinationSet, Map> sourceMap, + Map> destinationMap, + CodeUnitIterator sourceCodeUnitIterator, CodeUnitIterator destinationCodeUnitIterator) + throws CancelledException { + // get rid of the codeUnit we already have + if (sourceCodeUnitIterator.hasNext()) { + sourceCodeUnitIterator.next(); + } + + // get rid of the codeUnit we already have + if (destinationCodeUnitIterator.hasNext()) { + destinationCodeUnitIterator.next(); + } + + processLCS(sourceMap, destinationMap, sourceSet, destinationSet, sourceCodeUnitIterator, + destinationCodeUnitIterator, monitor); + } + + private void processLCS(Map> sourceMap, + Map> destinationMap, AddressSet sourceSet, + AddressSet destinationSet, CodeUnitIterator sourceCodeUnitIterator, + CodeUnitIterator destinationCodeUnitIterator, TaskMonitor monitor) + throws CancelledException { + List source = (sourceFunction != null) + ? getCodeUnits(sourceFunction, sourceSet, sourceCodeUnitIterator) + : new ArrayList(); + List destination = (destinationFunction != null) + ? getCodeUnits(destinationFunction, destinationSet, destinationCodeUnitIterator) + : new ArrayList(); + CodeUnitLCS culcs = new CodeUnitLCS(source, destination); + List lcs = culcs.getLcs(monitor); + final int lcsSize = lcs.size(); + int sourceII = 0; + int lcsII = 0; + int destinationII = 0; + + monitor.setMessage("Defining address ranges..."); + monitor.initialize(lcsSize); + + int sourceTransactionID = -1; + int destinationTransactionID = -1; + + try { + sourceTransactionID = + sourceFunction.getProgram().startTransaction("Colorize CodeCompare"); + destinationTransactionID = + destinationFunction.getProgram().startTransaction("Colorize CodeCompare"); + + while (lcsII < lcsSize) { + monitor.checkCancelled(); + CodeUnitContainer sourceCodeUnit = source.get(sourceII); + CodeUnitContainer lcsCodeUnit = lcs.get(lcsII); + CodeUnitContainer destinationCodeUnit = destination.get(destinationII); + final boolean sourceCompare = culcs.matches(sourceCodeUnit, lcsCodeUnit); + final boolean destinationCompare = culcs.matches(lcsCodeUnit, destinationCodeUnit); + if (sourceCompare == destinationCompare) { + // either they're both equal to lcs item or they're both different + if (sourceCompare) { + // they're both equal, define the ranges + defineRange(sourceMap, destinationMap, sourceCodeUnit, destinationCodeUnit); + // increment the lcs index because everything matched + ++lcsII; + } + // in any case, increment both the source and destination indexes + // because they were either both the same or both different + ++sourceII; + ++destinationII; + } + else if (sourceCompare) { + // destination has extra stuff (new code added) + ++destinationII; + } + else if (destinationCompare) { + // source has extra stuff (old code deleted) + ++sourceII; + } + else { + // can't get here! + throw new RuntimeException("internal error"); + } + + monitor.incrementProgress(1); + } + } + finally { + if (sourceTransactionID != -1) { + sourceFunction.getProgram().endTransaction(sourceTransactionID, true); + } + if (destinationTransactionID != -1) { + destinationFunction.getProgram().endTransaction(destinationTransactionID, true); + } + } + computeParamCorrelation(); + } + + protected void computeParamCorrelation() { + int sourceCount = sourceFunction.getParameterCount(); + int destinationCount = destinationFunction.getParameterCount(); + Parameter[] sourceParameters = sourceFunction.getParameters(); + Parameter[] destinationParameters = destinationFunction.getParameters(); + boolean allMatch = false; + Map map = new HashMap(); + if (sourceCount == destinationCount) { + allMatch = true; + for (int i = 0; i < sourceParameters.length; i++) { + Parameter sourceParameter = sourceParameters[i]; + Parameter destinationParameter = destinationParameters[i]; + if (!sourceParameter.isValid() || !destinationParameter.isValid() || + sourceParameter.getLength() != destinationParameter.getLength()) { + // - an invalid parameter does not have defined storage (or address) + // - length must also match + allMatch = false; + break; + } + Address dest = destinationParameter.getVariableStorage().getMinAddress(); + Address src = sourceParameter.getVariableStorage().getMinAddress(); + map.put(src, new CorrelationContainer(CorrelationKind.PARAMETERS, + new AddressRangeImpl(dest, dest))); + } + } + if (allMatch) { + cachedForwardAddressMap.putAll(map); + } + } + + private void defineRange(Map> sourceMap, + Map> destinationMap, CodeUnitContainer sourceCodeUnit, + CodeUnitContainer destinationCodeUnit) { + Address minAddress = sourceCodeUnit.getCodeUnit().getMinAddress(); + Address maxAddress = sourceCodeUnit.getCodeUnit().getMaxAddress(); + AddressRangeImpl toRange = + new AddressRangeImpl(destinationCodeUnit.getCodeUnit().getMinAddress(), + destinationCodeUnit.getCodeUnit().getMaxAddress()); + CorrelationContainer container = new CorrelationContainer(CorrelationKind.LCS, toRange); + + DebugUtils.recordEOLComment(sourceMap, sourceProgram, minAddress, maxAddress, + destinationProgram, destinationCodeUnit.getCodeUnit().getMinAddress(), + destinationCodeUnit.getCodeUnit().getMaxAddress()); + DebugUtils.recordEOLComment(destinationMap, destinationProgram, + destinationCodeUnit.getCodeUnit().getMinAddress(), + destinationCodeUnit.getCodeUnit().getMaxAddress(), sourceProgram, minAddress, + maxAddress); + + while (!minAddress.equals(maxAddress)) { + cachedForwardAddressMap.put(minAddress, container); + minAddress = minAddress.next(); + } + cachedForwardAddressMap.put(maxAddress, container); + } + + private static List getCodeUnits(Function function, + AddressSetView correlations, CodeUnitIterator codeUnits) { + AddressSetView body = function.getBody(); + ArrayList result = new ArrayList(); + while (codeUnits.hasNext()) { + CodeUnit next = codeUnits.next(); + Address address = next.getAddress(); + if (correlations.contains(address) || !body.contains(address)) { + break; + } + result.add(new CodeUnitContainer(next)); + } + return result; + } + + @Override + public String getName() { + return "CodeCompareAddressCorrelator"; + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/CodeCompareAddressCorrelator.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/CodeCompareAddressCorrelator.java new file mode 100755 index 0000000000..5d6a9f4afd --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/CodeCompareAddressCorrelator.java @@ -0,0 +1,68 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api.correlator.address; + +import ghidra.framework.options.Options; +import ghidra.framework.options.ToolOptions; +import ghidra.program.model.lang.Language; +import ghidra.program.model.listing.*; +import ghidra.program.util.AddressCorrelation; +import ghidra.program.util.DiscoverableAddressCorrelator; + +public class CodeCompareAddressCorrelator implements DiscoverableAddressCorrelator { + + private static final String OPTIONS_NAME = "CodeCompareAddressCorrelator"; + + private ToolOptions options = new ToolOptions(OPTIONS_NAME); + + public CodeCompareAddressCorrelator() { + } + + @Override + public synchronized AddressCorrelation correlate(Function sourceFunction, + Function destinationFunction) { + + Program p1 = sourceFunction.getProgram(); + Program p2 = destinationFunction.getProgram(); + Language l1 = p1.getLanguage(); + Language l2 = p2.getLanguage(); + if (l1.getLanguageID().equals(l2.getLanguageID())) { + return null; // this correlator is best used with different architectures + } + + return new CodeCompareAddressCorrelation(sourceFunction, destinationFunction); + } + + @Override + public AddressCorrelation correlate(Data sourceData, Data destinationData) { + return null; + } + + @Override + public ToolOptions getOptions() { + return options; + } + + @Override + public void setOptions(ToolOptions options) { + options = options.copy(); + } + + @Override + public Options getDefaultOptions() { + return new ToolOptions(OPTIONS_NAME); + } +} diff --git a/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/DebugUtils.java b/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/DebugUtils.java new file mode 100755 index 0000000000..504af48049 --- /dev/null +++ b/Ghidra/Features/CodeCompare/src/main/java/ghidra/feature/vt/api/correlator/address/DebugUtils.java @@ -0,0 +1,209 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api.correlator.address; + +import java.awt.Color; +import java.util.*; +import java.util.Map.Entry; + +import ghidra.app.util.viewer.listingpanel.PropertyBasedBackgroundColorModel; +import ghidra.feature.vt.api.correlator.address.CodeCompareAddressCorrelation.CorrelationContainer; +import ghidra.program.database.IntRangeMap; +import ghidra.program.model.address.*; +import ghidra.program.model.listing.*; +import ghidra.util.Msg; +import ghidra.util.exception.DuplicateNameException; + +class DebugUtils { + + private static boolean ENABLED = false; + + static void recordEOLComment(Map> map, Program fromProgram, + Address startAddress, Address endAddress, Program toProgram, Address minAddress, + Address maxAddress) { + + if (isEnabled()) { + Listing listing = fromProgram.getListing(); + AddressSet addrSet = new AddressSet(startAddress, endAddress); + CodeUnitIterator codeUnits = listing.getCodeUnits(addrSet, true); + AddressRange range = new AddressRangeImpl(minAddress, maxAddress); + + while (codeUnits.hasNext()) { + CodeUnit codeUnit = codeUnits.next(); + TreeSet set = map.get(codeUnit); + if (set == null) { + set = new TreeSet(); + map.put(codeUnit, set); + } + set.add(range); + } + } + } + + static void processMap(Map> map, Program program) { + if (isEnabled()) { + int transactionID = -1; + try { + transactionID = program.startTransaction("Colorize CodeCompare"); + Set>> entrySet = map.entrySet(); + for (Entry> entry : entrySet) { + entry.getKey().setComment(CodeUnit.EOL_COMMENT, entry.getValue().toString()); + } + } + finally { + if (transactionID != -1) { + program.endTransaction(transactionID, true); + } + } + } + } + + static void colorize(Map map, Program sourceProgram, + Program destinationProgram) { + if (isEnabled()) { + int sourceTransactionID = -1; + int destinationTransactionID = -1; + try { + Listing sourceListing = sourceProgram.getListing(); + Listing destinationListing = destinationProgram.getListing(); + + sourceTransactionID = sourceProgram.startTransaction("Colorize CodeCompare"); + destinationTransactionID = + destinationProgram.startTransaction("Colorize CodeCompare"); + + Set> entrySet = map.entrySet(); + for (Entry entry : entrySet) { + Address sourceAddress = entry.getKey(); + CorrelationContainer container = entry.getValue(); + Color color = pickColor(container); + colorCodeUnits(sourceProgram, color, getCodeUnit(sourceListing, sourceAddress)); + colorCodeUnits(destinationProgram, color, + getCodeUnits(destinationListing, container.range)); + } + } + finally { + if (sourceTransactionID != -1) { + sourceProgram.endTransaction(sourceTransactionID, true); + } + if (destinationTransactionID != -1) { + destinationProgram.endTransaction(destinationTransactionID, true); + } + } + } + } + + private static CodeUnit getCodeUnit(Listing listing, Address address) { + return listing.getCodeUnitContaining(address); + } + + private static CodeUnit[] getCodeUnits(Listing listing, AddressRange addressRange) { + HashSet units = new HashSet(); + Address address = addressRange.getMinAddress(); + Address maxAddress = addressRange.getMaxAddress(); + while (!address.equals(maxAddress)) { + CodeUnit codeUnit = listing.getCodeUnitContaining(address); + units.add(codeUnit); + try { + address = address.addNoWrap(1); + } + catch (AddressOverflowException e) { + Msg.debug(DebugUtils.class, "Woah...non-contiguous CodeBlock", e); + break; + } + } + CodeUnit codeUnit = listing.getCodeUnitContaining(maxAddress); + units.add(codeUnit); + return units.toArray(new CodeUnit[0]); + } + + private static void colorCodeUnits(Program program, Color color, CodeUnit... codeUnits) { + for (CodeUnit codeUnit : codeUnits) { + if (codeUnit != null) { + setBackgroundColor(program, codeUnit.getMinAddress(), codeUnit.getMaxAddress(), + color); + } + } + } + + private static void setBackgroundColor(Program program, Address min, Address max, Color c) { + IntRangeMap map = getColorRangeMap(program, true); + if (map != null) { + map.setValue(min, max, c.getRGB()); + } + } + + private static IntRangeMap getColorRangeMap(Program program, boolean create) { + if (program == null) { + return null; + } + IntRangeMap map = + program.getIntRangeMap(PropertyBasedBackgroundColorModel.COLOR_PROPERTY_NAME); + if (map == null && create) { + try { + map = program.createIntRangeMap( + PropertyBasedBackgroundColorModel.COLOR_PROPERTY_NAME); + } + catch (DuplicateNameException e) { + // can't happen since we just checked for it! + } + } + return map; + } + + private static Random RAND = new Random(); + + private static Color pickColor(CorrelationContainer container) { + float saturation; + float brightness; + float hue; + switch (container.kind) { + case CODE_COMPARE: + if (CodeCompareAddressCorrelation.CorrelationContainer.USE_RANDOM_CC_COLORS) { + hue = RAND.nextFloat(); + } + else { + hue = 0.33f; + } + saturation = (float) (0.5 - RAND.nextFloat() / 3.0); + brightness = (float) (1.0 - RAND.nextFloat() / 5.0); + break; + case LCS: + hue = 0.9f; + saturation = (float) (0.5 - RAND.nextFloat() / 3.0); + brightness = (float) (1.0 - RAND.nextFloat() / 5.0); + break; + case PARAMETERS: + hue = 0.2f; + saturation = (float) (1.0 - RAND.nextFloat() / 3.0); + brightness = (float) (1.0 - RAND.nextFloat() / 3.0); + break; + default: + hue = 0.1f; + saturation = 1.0f; + brightness = 1.0f; + break; + } + return Color.getHSBColor(hue, saturation, brightness); + } + + public static void enable(boolean b) { + ENABLED = b; + } + + private static boolean isEnabled() { + return ENABLED; + } +} diff --git a/Ghidra/Features/Decompiler/.gitignore b/Ghidra/Features/Decompiler/.gitignore deleted file mode 100644 index 72173fc677..0000000000 --- a/Ghidra/Features/Decompiler/.gitignore +++ /dev/null @@ -1,2 +0,0 @@ -# Ignore Doxygen documentation -*/*/doc diff --git a/Ghidra/Features/Decompiler/buildNatives.gradle b/Ghidra/Features/Decompiler/buildNatives.gradle index 45d312a797..9cf660e915 100644 --- a/Ghidra/Features/Decompiler/buildNatives.gradle +++ b/Ghidra/Features/Decompiler/buildNatives.gradle @@ -123,6 +123,8 @@ model { include "ghidra_process.cc" include "comment_ghidra.cc" include "string_ghidra.cc" + include "signature.cc" + include "signature_ghidra.cc" // include "callgraph.cc" // uncomment for debug // include "ifacedecomp.cc" // uncomment for debug // include "ifaceterm.cc" // uncomment for debug diff --git a/Ghidra/Features/Decompiler/certification.manifest b/Ghidra/Features/Decompiler/certification.manifest index 7933052f1a..eecf7c60c5 100644 --- a/Ghidra/Features/Decompiler/certification.manifest +++ b/Ghidra/Features/Decompiler/certification.manifest @@ -8,7 +8,6 @@ Module.manifest||GHIDRA||||END| data/decompiler.theme.properties||GHIDRA||||END| src/decompile/.cproject||GHIDRA||||END| -src/decompile/.project||GHIDRA||||END| src/decompile/cpp/.gitignore||GHIDRA||||END| src/decompile/cpp/Doxyfile||GHIDRA|||Most of this file is autogenerated by doxygen which falls under the GPL - output from GPL products are NOT GPL! - mjbell4|END| src/decompile/cpp/Makefile||GHIDRA||||END| diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/Makefile b/Ghidra/Features/Decompiler/src/decompile/cpp/Makefile index 64f3894d94..24d9bcd7ab 100644 --- a/Ghidra/Features/Decompiler/src/decompile/cpp/Makefile +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/Makefile @@ -84,14 +84,14 @@ DECCORE=capability architecture options graph cover block cast typeop database c funcdata funcdata_block funcdata_op funcdata_varnode unionresolve pcodeinject \ heritage prefersplit rangeutil ruleaction subflow blockaction merge double \ transform coreaction condexe override dynamic crc32 prettyprint \ - printlanguage printc printjava memstate opbehavior paramid $(COREEXT_NAMES) + printlanguage printc printjava memstate opbehavior paramid signature $(COREEXT_NAMES) # Files used for any project that use the sleigh decoder SLEIGH= sleigh pcodeparse pcodecompile sleighbase slghsymbol \ slghpatexpress slghpattern semantics context filemanage # Additional files for the GHIDRA specific build GHIDRA= ghidra_arch inject_ghidra ghidra_translate loadimage_ghidra \ typegrp_ghidra database_ghidra ghidra_context cpool_ghidra \ - ghidra_process comment_ghidra string_ghidra $(GHIDRAEXT_NAMES) + ghidra_process comment_ghidra string_ghidra signature_ghidra $(GHIDRAEXT_NAMES) # Additional files specific to the sleigh compiler SLACOMP=slgh_compile slghparse slghscan # Additional special files that should not be considered part of the library diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/analyzesigs.cc b/Ghidra/Features/Decompiler/src/decompile/cpp/analyzesigs.cc new file mode 100755 index 0000000000..d3c2bb4b36 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/analyzesigs.cc @@ -0,0 +1,243 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "analyzesigs.hh" +#include "loadimage_bfd.hh" + +namespace ghidra { + +// Constructing this registers the capability +IfaceAnalyzeSigsCapability IfaceAnalyzeSigsCapability::ifaceAnalyzeSigsCapability; + +IfaceAnalyzeSigsCapability::IfaceAnalyzeSigsCapability(void) + +{ + name = "analyzesigs"; +} + +void IfaceAnalyzeSigsCapability::registerCommands(IfaceStatus *status) + +{ + status->registerCom(new IfcSignatureSettings(), "signature", "settings"); + status->registerCom(new IfcPrintSignatures(),"print","signatures"); + status->registerCom(new IfcSaveSignatures(),"save","signatures"); + status->registerCom(new IfcSaveAllSignatures(),"saveall","signatures"); + status->registerCom(new IfcProduceSignatures(),"produce","signatures"); +} + +/// \class IfcSignatureSettings +/// \brief Change global settings for signature generation : `signature settings ` +/// +/// The provided integer value establishes the settings for any future signature generation +void IfcSignatureSettings::execute(istream &s) + +{ + uint4 mysetting = 0; + + s.unsetf(ios::dec | ios::hex | ios::oct); // Let user specify base + s >> mysetting; + if (mysetting == 0) + throw IfaceParseError("Must specify settings integer"); + SigManager::setSettings(mysetting); + *status->optr << "Signature settings set to " << hex << mysetting << endl; +} + +/// \class IfcPrintSignatures +/// \brief Calculate and print signatures for the current function: `print signatures [...]` +/// +/// Decompilation must already be complete. Features are extracted from the function and are +/// printed, one per line. The command optionally takes additional parameters that can alter +/// signature generation. +void IfcPrintSignatures::execute(istream &s) + +{ // + if (dcp->fd == (Funcdata *)0) + throw IfaceExecutionError("No function selected"); + if (!dcp->fd->isProcComplete()) + throw IfaceExecutionError("Function has not been fully analyzed"); + + GraphSigManager smanage; + + smanage.initializeFromStream(s); + + *status->fileoptr << "Signatures for " << dcp->fd->getName() << endl; + + smanage.setCurrentFunction(dcp->fd); + smanage.generate(); + smanage.print(*status->fileoptr); +} + +/// \class IfcSaveSignatures +/// \brief Calculate signatures and save them to a file: `save signatures [...]` +/// +/// The features/signatures are extracted from the current function, which must already be +/// decompiled, and are written out in XML format. The first parameter must be the file name. +/// The command optionally takes additional parameters that can alter signature generation. +void IfcSaveSignatures::execute(istream &s) + +{ + if (dcp->fd == (Funcdata *)0) + throw IfaceExecutionError("No function selected"); + if (!dcp->fd->isProcComplete()) + throw IfaceExecutionError("Function has not been fully analyzed"); + + string sigfilename; + + s >> sigfilename; + if (sigfilename.size()==0) + throw IfaceExecutionError("Need name of file to save signatures to"); + + GraphSigManager smanage; + smanage.initializeFromStream(s); + + smanage.setCurrentFunction(dcp->fd); + smanage.generate(); + ofstream t( sigfilename.c_str() ); + + if (!t) + throw IfaceExecutionError("Unable to open signature save file: "+sigfilename); + + XmlEncode encoder(t); + smanage.encode(encoder); + t.close(); + + *status->fileoptr << "Successfully saved signatures for " << dcp->fd->getName() << endl; +} + +/// \class IfcSaveAllSignatures +/// \brief Calculate signatures and save them to a file: `saveall signatures [...]` +/// +/// For every known function entry point, the function is decompiled (using the current action) +/// and features/signatures are extracted. Features are written out in XML format to the +/// file indicated by the first parameter. The command optionally takes additional parameters +/// that can alter signature generation. +void IfcSaveAllSignatures::execute(istream &s) + +{ + if (dcp->conf == (Architecture *)0) + throw IfaceExecutionError("No architecture loaded"); + + string sigfilename; + + s >> sigfilename; + if (sigfilename.size() == 0) + throw IfaceExecutionError("Need name of file to save signatures to"); + + if (smanage != (GraphSigManager *)0) + delete smanage; + smanage = new GraphSigManager(); + smanage->initializeFromStream(s); // configure the manager; + + ostream *saveoldfileptr = status->fileoptr; + status->fileoptr = new ofstream; + ((ofstream *)status->fileoptr)->open(sigfilename.c_str()); + if (!*status->fileoptr) { + delete status->fileoptr; + status->fileoptr = saveoldfileptr; + throw IfaceExecutionError("Unable to open signature save file: "+sigfilename); + } + + string oldactname = dcp->conf->allacts.getCurrentName(); + dcp->conf->allacts.setCurrent("normalize"); + iterateFunctionsAddrOrder(); + + ((ofstream *)status->fileoptr)->close(); + delete status->fileoptr; + status->fileoptr = saveoldfileptr; + + dcp->conf->allacts.setCurrent(oldactname); + delete smanage; + smanage = (GraphSigManager *)0; +} + +void IfcSaveAllSignatures::iterationCallback(Funcdata *fd) + +{ + if (fd->hasNoCode()) { + *status->optr << "No code for " << fd->getName() << endl; + return; + } + try { + dcp->conf->clearAnalysis(fd); // Clear any old analysis + dcp->conf->allacts.getCurrent()->reset(*fd); + dcp->conf->allacts.getCurrent()->perform( *fd ); + *status->optr << "Decompiled " << fd->getName(); + *status->optr << '(' << dec << fd->getSize() << ')' << endl; + } + catch(LowlevelError &err) { + *status->optr << "Skipping " << fd->getName() << ": " << err.explain << endl; + return; + } + + smanage->setCurrentFunction(fd); + smanage->generate(); + + uint4 numsigs = smanage->numSignatures(); + if (numsigs != 0) { + Address addr = fd->getAddress(); + uint4 spcindex = addr.getSpace()->getIndex(); + uintb off = addr.getOffset(); + status->fileoptr->write((char *)&spcindex,4); + status->fileoptr->write((char *)&off,sizeof(uintb)); + status->fileoptr->write((char *)&numsigs,4); + uint4 namelen = fd->getName().size(); + status->fileoptr->write((char *)&namelen,4); + status->fileoptr->write(fd->getName().c_str(),namelen); + XmlEncode encoder(*status->fileoptr); + smanage->encode(encoder); + } + smanage->clear(); + + dcp->conf->clearAnalysis(fd); +} + +/// \class IfcProduceSignatures +/// \brief Calculate signatures and save combined hashes to a file: `produce signatures [...]` +/// +/// For every known function entry point, the function is decompiled (using the current action) +/// and features/signatures are extracted. Features for a single function are combined using an +/// overall hash and written out to the file indicated by the first parameter. The file will contain +/// one line per function, with the name of the function followed by the overall hash. The command +/// optionally takes additional parameters that can alter signature generation. +void IfcProduceSignatures::iterationCallback(Funcdata *fd) + +{ + if (fd->hasNoCode()) { + *status->optr << "No code for " << fd->getName() << endl; + return; + } + try { + dcp->conf->clearAnalysis(fd); // Clear any old analysis + dcp->conf->allacts.getCurrent()->reset(*fd); + dcp->conf->allacts.getCurrent()->perform( *fd ); + *status->optr << "Decompiled " << fd->getName(); + *status->optr << '(' << dec << fd->getSize() << ')' << endl; + } + catch(LowlevelError &err) { + *status->optr << "Skipping " << fd->getName() << ": " << err.explain << endl; + return; + } + + smanage->setCurrentFunction(fd); + smanage->generate(); + hashword finalsig = smanage->getOverallHash(); + (*status->fileoptr) << fd->getName() << " = 0x" << hex << setfill('0') << setw(16) << finalsig << endl; + + smanage->clear(); + + dcp->conf->clearAnalysis(fd); +} + +} // End namespace ghidra diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/analyzesigs.hh b/Ghidra/Features/Decompiler/src/decompile/cpp/analyzesigs.hh new file mode 100755 index 0000000000..33d0857bc7 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/analyzesigs.hh @@ -0,0 +1,67 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +/// \file analyzesigs.hh +/// \brief Commands for feature/signature generation from the console interface +#ifndef __ANALYZESIGS_HH__ +#define __ANALYZESIGS_HH__ + +#include "codedata.hh" +#include "signature.hh" + +namespace ghidra { + +/// \brief Interface capability point for console commands associated with signature/feature generation +class IfaceAnalyzeSigsCapability : public IfaceCapability { + static IfaceAnalyzeSigsCapability ifaceAnalyzeSigsCapability; ///< Singleton instance + IfaceAnalyzeSigsCapability(void); ///< Construct the singleton + IfaceAnalyzeSigsCapability(const IfaceAnalyzeSigsCapability &op2); ///< Not implemented + IfaceAnalyzeSigsCapability &operator=(const IfaceAnalyzeSigsCapability &op2); ///< Not implemented +public: + virtual void registerCommands(IfaceStatus *status); +}; + +class IfcSignatureSettings : public IfaceDecompCommand { +public: + virtual void execute(istream &s); +}; + +class IfcPrintSignatures : public IfaceDecompCommand { +public: + virtual void execute(istream &s); +}; + +class IfcSaveSignatures : public IfaceDecompCommand { +public: + virtual void execute(istream &s); +}; + +class IfcSaveAllSignatures : public IfaceDecompCommand { +protected: + GraphSigManager *smanage; ///< Manager for generating signatures +public: + IfcSaveAllSignatures(void) { smanage = (GraphSigManager *)0; } ///< Constructor + virtual ~IfcSaveAllSignatures(void) { if (smanage != (GraphSigManager *)0) delete smanage; } + virtual void execute(istream &s); + virtual void iterationCallback(Funcdata *fd); +}; + +class IfcProduceSignatures : public IfcSaveAllSignatures { +public: + virtual void iterationCallback(Funcdata *fd); +}; + +} // End namespace ghidra +#endif diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/architecture.cc b/Ghidra/Features/Decompiler/src/decompile/cpp/architecture.cc index d4276a8a0e..2ec69d21ad 100644 --- a/Ghidra/Features/Decompiler/src/decompile/cpp/architecture.cc +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/architecture.cc @@ -32,7 +32,7 @@ using std::sqrt; vector ArchitectureCapability::thelist; -const uint4 ArchitectureCapability::majorversion = 5; +const uint4 ArchitectureCapability::majorversion = 6; const uint4 ArchitectureCapability::minorversion = 0; AttributeId ATTRIB_ADDRESS = AttributeId("address",148); diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/signature.cc b/Ghidra/Features/Decompiler/src/decompile/cpp/signature.cc new file mode 100755 index 0000000000..1ce1a3dc93 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/signature.cc @@ -0,0 +1,1148 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "signature.hh" +#include "crc32.hh" + +namespace ghidra { + +uint4 SigManager::settings = 0; // User MUST set a valid settings +// The current suggested defaults +// ((GraphSigManager::SIG_DONOTUSE_CONST | +// GraphSigManager::SIG_COLLAPSE_INDNOISE)<<2)|1 + +AttributeId ATTRIB_BADDATA = AttributeId("baddata",145); +AttributeId ATTRIB_HASH = AttributeId("hash",146); +AttributeId ATTRIB_UNIMPL = AttributeId("unimpl",147); + +ElementId ELEM_BLOCKSIG = ElementId("blocksig",258); +ElementId ELEM_CALL = ElementId("call",259); +ElementId ELEM_GENSIG = ElementId("gensig",260); +ElementId ELEM_MAJOR = ElementId("major",261); +ElementId ELEM_MINOR = ElementId("minor",262); +ElementId ELEM_COPYSIG = ElementId("copysig",263); +ElementId ELEM_SETTINGS = ElementId("settings",264); +ElementId ELEM_SIG = ElementId("sig",265); +ElementId ELEM_SIGNATUREDESC = ElementId("signaturedesc",266); +ElementId ELEM_SIGNATURES = ElementId("signatures",267); +ElementId ELEM_SIGSETTINGS = ElementId("sigsettings",268); +ElementId ELEM_VARSIG = ElementId("varsig",269); + +static hashword hash_mixin(hashword val1,hashword val2) + +{ + uint4 hashhi = (uint4)(val1>>32); + uint4 hashlo = (uint4)val1; + for(int4 i=0;i<8;++i) { + uint4 tmphi = hashhi; + uint4 tmplo = (uint4)val2; + val2 >>= 8; + hashhi = crc_update(hashhi,tmplo); + hashlo = crc_update(hashlo,tmphi); + } + uint8 res = hashhi; + res <<= 32; + res |= hashlo; + return res; +} + +/// \param s is the given stream to write to +void Signature::print(ostream &s) const + +{ + s << '*'; + printOrigin(s); + s << " = 0x" << hex << setw(8) << setfill('0') << sig << endl; +} + +/// The underlying hashes of the two features are compared as unsigned values. +/// \param op2 is the other feature to compare with \b this +/// \return -1, 0, or 1 if \b this is ordered before, equal to, or after the other feature +int4 Signature::compare(const Signature *op2) const + +{ + if (sig != op2->sig) + return (sig < op2->sig) ? -1 : 1; + return 0; +} + +/// A Varnode shadows another if it is defined by a COPY or INDIRECT op. In this case, the +/// \b shadow field is filled in by following any chain of COPY and INDIRECT ops back to their input Varnode. +/// \param sigMap is the map from Varnode to its SignatureEntry overlay +void SignatureEntry::calculateShadow(const map &sigMap) + +{ + const Varnode *shadowVn = vn; + for(;;) { + op = shadowVn->getDef(); + if (op == (const PcodeOp *)0) + break; + OpCode opc = op->code(); + if (opc != CPUI_COPY && opc != CPUI_INDIRECT && opc != CPUI_CAST) + break; + shadowVn = op->getIn(0); + } + if (shadowVn != vn) + shadow = mapToEntry(shadowVn,sigMap); +} + +/// In most cases, the hash value is just the OpCode of the PcodeOp itself. +/// CPOOLREF ops are distinguished by hashing in the CPoolRecord tag type. +/// \param modifiers are the specific settings being used for signature generation +/// \return the PcodeOp hash value or 0 if there is no effective PcodeOp +hashword SignatureEntry::getOpHash(uint4 modifiers) + +{ + if (op == (const PcodeOp *)0) + return 0; + OpCode opc = op->code(); + hashword ophash = (hashword)opc; + // For constant pool operations, hash in the resolved tag type constant, which is the last input + if (opc == CPUI_CPOOLREF) + ophash = (ophash + 0xfeedface) ^ op->getIn(op->numInput()-1)->getOffset(); + return ophash; +} + +/// A stand-alone COPY is incorporated as a feature differently from other Varnodes. +/// This method computes its hash once up front, and it is not updated iteratively as +/// with other SignatureEntry nodes. +/// \param modifiers are the specific settings being used for signature generation +void SignatureEntry::standaloneCopyHash(uint4 modifiers) + +{ + hashword val = hashSize(vn,modifiers); + val ^= 0xaf29e23b; + if (vn->isPersist()) + val ^= (hashword) 0x55055055; + Varnode *invn = vn->getDef()->getIn(0); + if (invn->isConstant()) { + if ((modifiers&GraphSigManager::SIG_DONOTUSE_CONST)==0) + val ^= vn->getOffset(); + else + val ^= 0xa0a0a0a0; + } + else if (invn->isPersist()) + val ^= 0xd7651ec3; + hash[0] = val; + hash[1] = val; +} + +/// \param v is the Varnode being overlayed +/// \param modifiers are the settings being used for signature generation +SignatureEntry::SignatureEntry(Varnode *v,uint4 modifiers) + +{ + vn = v; + op = vn->getDef(); + inSize=0; + flags=0; + shadow = (SignatureEntry *)0; + index = -1; + + // Decide on the effective defining op for the given varnode. + if (op == (const PcodeOp *)0) { + flags |= SIG_NODE_TERMINAL; + return; + } + startvn = 0; + inSize = op->numInput(); + switch(op->code()) { + case CPUI_COPY: + if (testStandaloneCopy(vn)) + flags |= SIG_NODE_STANDALONE; + break; + case CPUI_INDIRECT: + inSize -= 1; + if (testStandaloneCopy(vn)) + flags |= SIG_NODE_STANDALONE; + break; + case CPUI_MULTIEQUAL: + flags |= SIG_NODE_COMMUTATIVE; + break; + case CPUI_CALL: + startvn = 1; + inSize -= 1; + break; + case CPUI_CALLIND: + startvn = 1; + inSize -= 1; + break; + case CPUI_CALLOTHER: + startvn = 1; + inSize -= 1; + break; + case CPUI_STORE: + startvn = 1; + inSize -= 1; + break; + case CPUI_LOAD: + startvn = 1; + inSize -= 1; + break; + case CPUI_INT_LEFT: + case CPUI_INT_RIGHT: + case CPUI_INT_SRIGHT: // For shift + case CPUI_SUBPIECE: // and truncation operators + if (op->getIn(1)->isConstant()) // if the shift/truncation amount is constant + inSize = 1; // make sure we don't even hash in the size of the constant + break; + case CPUI_CPOOLREF: + inSize = 0; // Only hash-in first input + break; + default: + if (op->isCommutative()) + flags |= SIG_NODE_COMMUTATIVE; + break; + } +} + +/// Used for adding additional virtual nodes in a modified graph. There is no actual backing Varnode. +/// \param ind is the post-order index to associate with the virtual node +SignatureEntry::SignatureEntry(int4 ind) + +{ + vn = (Varnode *)0; + op = (PcodeOp *)0; + inSize=0; + flags=0; + shadow = (SignatureEntry *)0; + index = ind; + startvn = 0; +} + +/// A Varnode is a stand-alone COPY if it is defined by a COPY or INDIRECT operation whose +/// input Varnode is either a constant or an input. Additionally the Varnode must not be +/// read directly by other operations in the function. There currently is a slight exception +/// made for a \e persistent Varnode; it can be read through a single INDIRECT operation. +/// \param vn is the given Varnode to test +/// \return \b true if it is considered a stand-alone COPY +bool SignatureEntry::testStandaloneCopy(Varnode *vn) + +{ + PcodeOp *op = vn->getDef(); + const Varnode *invn = op->getIn(0); + if (invn->isWritten()) + return false; + if (invn->getAddr() == vn->getAddr()) + return false; + + if (vn->isPersist() && op->code() == CPUI_INDIRECT) + return true; + list::const_iterator iter = vn->beginDescend(); + if (iter == vn->endDescend()) + return true; + PcodeOp *descOp = *iter; + ++iter; + if (iter != vn->endDescend()) + return false; + OpCode opc = descOp->code(); + if (vn->isPersist() && opc == CPUI_INDIRECT) { + return true; + } + // Account for COPY and INDIRECT placeholder conventions + if (opc != CPUI_COPY && opc != CPUI_INDIRECT) + return false; + return descOp->getOut()->hasNoDescend(); +} + +/// The current hash value is set, incorporating: +/// - the size of the Varnode +/// - whether the Varnode is constant +/// - whether the Varnode is an input +/// - whether the Varnode is persistent +/// - the OpCode of the effective defining PcodeOp +/// +/// \param modifiers are the settings being used for signature generation +void SignatureEntry::localHash(uint4 modifiers) + +{ + hashword localhash; + + if (vn->isAnnotation()) { + localhash = 0xb7b7b7b7; // This mostly won't get used, but in some rare cases + flags |= (SIG_NODE_NOT_EMITTED | SIG_NODE_TERMINAL); + hash[0] = localhash; // Allow terminal node to access value from both slots without moving data in iteration + hash[1] = localhash; + return; + } + if (shadow != (SignatureEntry *)0) { // If this is a shadow + flags |= SIG_NODE_NOT_EMITTED; // Don't emit as a feature + if (isStandaloneCopy()) { + standaloneCopyHash(modifiers); // Standalone COPY gets special hash + } + return; // Don't calculate hash otherwise + } + + localhash = hashSize(vn,modifiers); + + if (!vn->isWritten()) // If there isn't at least some tree above this varnode + flags |= SIG_NODE_NOT_EMITTED; // Don't emit this hash as a feature, but still generate hash + hashword ophash = getOpHash(modifiers); + + // Class of varnode + if (vn->isConstant()) { + if ((modifiers&GraphSigManager::SIG_DONOTUSE_CONST)==0) + localhash ^= vn->getOffset(); + else + localhash ^= 0xa0a0a0a0; + } + // Note that internally, a persist storage location may get propagated away + // so make sure varnode is an input. + if ((modifiers&GraphSigManager::SIG_DONOTUSE_PERSIST)==0) { + if (vn->isPersist()&&vn->isInput()) + localhash ^= (hashword) 0x55055055; + } +// if ((modifiers&GraphSigManager::SIG_DONOTUSE_INPUT)==0) + if (vn->isInput()) + localhash ^= (hashword) 0x10101; + if (ophash != 0) { + localhash ^= ophash ^ (ophash<<9) ^ (ophash<<18); + } + + hash[0] = localhash; // Allow terminal nodes to access value from both slots without moving data in iteration + hash[1] = localhash; +} + +/// The previous hash value for \b this node is mixed together with previous hash values from other given nodes. +/// The result becomes the current hash value for \b this node. +/// \param neigh is the list of given nodes to mix in +void SignatureEntry::hashIn(vector &neigh) + +{ + hashword curhash = hash[1]; + if (isCommutative()) { + hashword accum = 0; + hashword tmphash; + for(int4 i=0;ihash[1]); + accum += tmphash; + } + curhash = hash_mixin(curhash,accum); + } + else { + for(int4 i=0;ihash[1]); + } + } + hash[0] = curhash; +} + +/// \brief Do a post-ordering of the modified \e noise graph +/// +/// The \e noise graph is formed from the original graph by removing all non-marker edges. +/// Root Varnodes for the \e noise graph, which are either inputs or are defined by a non-marker operation +/// in the original graph, must be provided. The SignatureEntry objects are passed back in post-order, and the +/// post-order index is stored on each object. +/// \param rootlist contains the roots of the graph +/// \param postOrder will hold the array of SignatureEntrys in post-order +/// \param sigMap is the map from Varnode to its SignatureEntry overlay +void SignatureEntry::noisePostOrder(const vector &rootlist,vector &postOrder,map &sigMap) + +{ + vector stack; + for(int4 i=0;ivn->beginDescend(); + entry->setVisited(); + while(!stack.empty()) { + entry = stack.back().entry; + list::const_iterator iter = stack.back().iter; + if (iter == entry->vn->endDescend()) { // No more children to traverse + stack.pop_back(); + entry->index = postOrder.size(); + postOrder.push_back(entry); + } + else { + PcodeOp *op = *iter; + stack.back().iter = ++iter; + if (op->isMarker() || op->code() == CPUI_COPY) { + SignatureEntry *childEntry = mapToEntry(op->getOut(),sigMap); + if (!childEntry->isVisited()) { + childEntry->setVisited(); + stack.push_back(DFSNode()); + stack.back().entry = childEntry; + stack.back().iter = childEntry->vn->beginDescend(); + } + } + } + } + } +} + +/// \brief Construct the dominator tree for the modified \e noise graph +/// +/// The \e noise graph is formed from the original graph by removing all non-marker edges. +/// Additionally a virtual root links all Varnodes that were inputs or had their defining op removed. +/// After this routine completes, the shadow field of each node is filled in with its immediate dominator +/// relative to the modified \e noise graph. +/// \param postOrder is the list of nodes in the \e noise graph in post-order +/// \param sigMap is the map from Varnode to its SignatureEntry overlay +void SignatureEntry::noiseDominator(vector &postOrder,map &sigMap) + +{ + SignatureEntry *b,*virtualRoot; + virtualRoot = b = postOrder.back(); // The official start node + b->shadow = b; + bool changed = true; + SignatureEntry *new_idom = (SignatureEntry *)0; + while(changed) { + changed = false; + for(int4 i=postOrder.size()-2;i>=0;--i) { // For all nodes, in reverse post-order, except root + b = postOrder[i]; + if (b->shadow != postOrder.back()) { + int4 j; + int4 sizeIn = b->markerSizeIn(); + for(j=0;jgetMarkerIn(j,virtualRoot,sigMap); + if (new_idom->shadow != (SignatureEntry *)0) + break; + } + j += 1; + for(;jgetMarkerIn(j,virtualRoot,sigMap); + if (rho->shadow != (SignatureEntry *)0) { // Here is the intersection routine + int4 finger1 = rho->index; + int4 finger2 = new_idom->index; + while(finger1 != finger2) { + while(finger1 < finger2) + finger1 = postOrder[finger1]->shadow->index; + while(finger2 < finger1) + finger2 = postOrder[finger2]->shadow->index; + } + new_idom = postOrder[finger1]; + } + } + if (b->shadow != new_idom) { + b->shadow = new_idom; + changed = true; + } + } + } + } +} + +/// \brief Remove \e noise from the data-flow graph by collapsing Varnodes that are indirect copies of each other +/// +/// Two Varnodes are indirect copies if the only paths from one to the other are made up of: +/// - CPUI_COPY +/// - CPUI_INDIRECT and +/// - CPUI_MULTIEQUAL +/// +/// This routine creates a \e noise (sub-)graph from the data-flow graph by removing all PcodeOp edges +/// that are not one of these three. A virtual root is created for the \e noise graph connecting all +/// the input Varnodes and Varnodes defined by one of the removed PcodeOps. The dominator tree +/// is calculated for this rooted \e noise graph. Any Varnode whose immediate dominator is the virtual root +/// is a \e shadow \e base and is not an indirect copy of any other Varnode. All other Varnodes are +/// indirect copies of one of these shadow bases, which can be calculated from the dominator tree. +/// Indirectly copied varnodes are collapsed into their shadow base and will have a non-zero \b shadow +/// field pointing to this base. +/// \param sigMap is the map from Varnode to its SignatureEntry overlay +void SignatureEntry::removeNoise(map &sigMap) + +{ + vector rootlist; + vector postOrder; + + // Set up the virtual root, by calculating all the nodes it is connected to into -rootlist- + map::const_iterator iter; + for(iter=sigMap.begin();iter!=sigMap.end();++iter) { + SignatureEntry *entry = (*iter).second; + Varnode *vn = entry->vn; + if (vn->isInput() || vn->isConstant()) { // Varnode is an input OR + rootlist.push_back(entry); + entry->flags |= MARKER_ROOT; + } + else if (vn->isWritten()) { + PcodeOp *op = vn->getDef(); + if ((!op->isMarker()) && op->code() != CPUI_COPY) { // Varnode is defined by a non-marker + rootlist.push_back(entry); + entry->flags |= MARKER_ROOT; + } + } + } + + noisePostOrder(rootlist,postOrder,sigMap); + // To get a proper unique root for the dominator algorithm, + // we construct a virtual root with out edges to every node in -rootlilst- + // -postOrder- is the list of nodes already in forward post-order + SignatureEntry virtualRoot(postOrder.size()); + postOrder.push_back(&virtualRoot); + for(int4 i=0;ishadow = &virtualRoot; // connects to (to deal with possible artificial edge) + + noiseDominator(postOrder,sigMap); + postOrder.pop_back(); // Pop off virtual root + + // Calculate the shadow bases and set their -shadow- field to null + for(int4 i=0;ishadow == &virtualRoot) + entry->shadow = (SignatureEntry *)0; + } + // Set the final shadow field by collapsing the dominator tree to the shadow bases + for(int4 i=0;ishadow != (SignatureEntry *)0) { + base = base->shadow; + } + while(entry->shadow != (SignatureEntry *)0) { + SignatureEntry *tmp = entry; + entry = entry->shadow; + tmp->shadow = base; + } + } +} + +#ifdef COPYNOISE_DEBUG +/// Check if \b this should have a \b shadow or not. +/// If so, make sure all inputs to \b this share the same shadow. +/// \param sigMap is the map from Varnode to its SignatureEntry overlay +void SignatureEntry::verifyNoiseRemoval(map &sigMap) const + +{ + if (shadow == (SignatureEntry *)0) { + if (!vn->isWritten()) return; + PcodeOp *op = vn->getDef(); + OpCode opc = op->code(); + if (opc == CPUI_COPY || opc == CPUI_INDIRECT) + throw LowlevelError("Node should be shadowed but isnt"); + return; + } + if (!vn->isWritten()) + throw LowlevelError("Shadowed node has no input"); + PcodeOp *op = vn->getDef(); + OpCode opc = op->code(); + if (opc == CPUI_COPY || opc == CPUI_INDIRECT) { + Varnode *invn = op->getIn(0); + SignatureEntry *inEntry = sigMap[invn->getCreateIndex()]; + if (inEntry->shadow == (SignatureEntry *)0) { // Terminal point of COPY chain + if (shadow != inEntry) + throw LowlevelError("Shadow does not match terminator"); + } + else if (inEntry->shadow != shadow) + throw LowlevelError("Shadow mismatch between varnode and COPY/INDIRECT input"); + } + else if (opc == CPUI_MULTIEQUAL) { + for(int4 i=0;inumInput();++i) { + Varnode *invn = op->getIn(i); + SignatureEntry *inEntry = sigMap[invn->getCreateIndex()]; + if (inEntry->shadow == (SignatureEntry *)0) { // Terminal point of COPY chain + if (shadow != inEntry) + throw LowlevelError("Shadow does not match multi terminator"); + } + else if (inEntry->shadow != shadow) + throw LowlevelError("Shadow mismatch between varnode and MULTIEQUAL input"); + } + } + else + throw LowlevelError("Shadowing varnode not written by COPY/INDIRECT/MULTIEQUAL"); +} + +/// Run through all SignatureEntry in the map. +/// \param sigMap is the map from Varnode to its SignatureEntry overlay +void SignatureEntry::verifyAllNoiseRemoval(map &sigMap) + +{ + map::const_iterator iter; + for(iter=sigMap.begin();iter!=sigMap.end();++iter) { + (*iter).second->verifyNoiseRemoval(sigMap); + } +} +#endif + +/// The current hash value is set, incorporating the number of incoming edges and the number of outgoing +/// edges, as integer values. +/// \param modifiers are the settings being used for signature generation +void BlockSignatureEntry::localHash(uint4 modifiers) + +{ + hashword localhash = bl->sizeIn(); + localhash <<= 8; + localhash |= bl->sizeOut(); + hash[0] = localhash; +} + +/// The previous hash value for \b this node is mixed together with previous hash values from other given nodes. +/// The result becomes the current hash value for \b this node. +/// The given nodes \e must correspond one-to-one with incoming blocks of \b this. +/// \param neigh is the list of nodes coming in to \b this +void BlockSignatureEntry::hashIn(vector &neigh) + +{ + hashword curhash = hash[1]; + hashword accum = 0xbafabaca; + hashword tmphash; + for(int4 i=0;ihash[1] ); + if (entry->bl->sizeOut() == 2) { // If the incoming block ends in a CBRANCH + if (bl->getInRevIndex(i) == 0) + tmphash = hash_mixin(tmphash,0x777 ^ 0x7abc7abc); // Incoming edge on FALSE condition + else + tmphash = hash_mixin(tmphash,0x777); // Incoming edge on TRUE condition + } + accum += tmphash; + } + hash[0] = hash_mixin(curhash,accum); +} + +/// The hash value is encoded to the stream, along with any descriptive information about +/// how the feature was formed. +/// \param encoder is the stream encoder +void Signature::encode(Encoder &encoder) const + +{ + encoder.openElement(ELEM_GENSIG); + encoder.writeUnsignedInteger(ATTRIB_HASH, getHash()); + encoder.closeElement(ELEM_GENSIG); +} + +/// The hash value corresponding to \b this feature is read from the stream. +/// \param decoder is the stream decoder +void Signature::decode(Decoder &decoder) + +{ + uint4 elemId = decoder.openElement(ELEM_GENSIG); + sig = decoder.readUnsignedInteger(ATTRIB_HASH); + decoder.closeElement(elemId); +} + +void VarnodeSignature::encode(Encoder &encoder) const + +{ + encoder.openElement(ELEM_VARSIG); + encoder.writeUnsignedInteger(ATTRIB_HASH, getHash()); + vn->encode(encoder); + if (vn->isWritten()) + vn->getDef()->encode(encoder); + encoder.closeElement(ELEM_VARSIG); +} + +void BlockSignature::encode(Encoder &encoder) const + +{ + encoder.openElement(ELEM_BLOCKSIG); + encoder.writeUnsignedInteger(ATTRIB_HASH, getHash()); + encoder.writeSignedInteger(ATTRIB_INDEX, bl->getIndex()); + bl->getStart().encode(encoder); + if (op2 != (const PcodeOp *)0) + op2->encode(encoder); + if (op1 != (const PcodeOp *)0) + op1->encode(encoder); + encoder.closeElement(ELEM_BLOCKSIG); +} + +void CopySignature::encode(Encoder &encoder) const + +{ + encoder.openElement(ELEM_COPYSIG); + encoder.writeUnsignedInteger(ATTRIB_HASH, getHash()); + encoder.writeSignedInteger(ATTRIB_INDEX, bl->getIndex()); + encoder.closeElement(ELEM_COPYSIG); +} + +void CopySignature::printOrigin(ostream &s) const + +{ + s << "Copies in "; + bl->printHeader(s); +} + +/// Clear any Signature objects specifically +void SigManager::clearSignatures(void) + +{ + for(int4 i=0;i &feature) const + +{ + feature.resize(sigs.size(),0); + for(uint4 i=0;igetHash(); + sort(feature.begin(),feature.end()); +} + +/// \return the overall hash value +hashword SigManager::getOverallHash(void) const + +{ + vector feature; + getSignatureVector(feature); + hashword pool = 0x12349876abacab; + for(uint4 i=0;i::const_iterator iter; + for(iter=sigs.begin();iter!=sigs.end();++iter) + (*iter)->print(s); +} + +/// Full details about all features currently stored in \b this manager are written to the stream. +/// \param encoder is the stream encoder +void SigManager::encode(Encoder &encoder) const + +{ + encoder.openElement(ELEM_SIGNATUREDESC); + vector::const_iterator iter; + for(iter=sigs.begin();iter!=sigs.end();++iter) + (*iter)->encode(encoder); + encoder.closeElement(ELEM_SIGNATUREDESC); +} + +/// \param newvalue are the settings to be used +void SigManager::setSettings(uint4 newvalue) + +{ + settings = newvalue; +} + +/// The (previously computed) final hash value for all Varnodes are emitted as VarnodeSignature features. +void GraphSigManager::collectVarnodeSigs(void) + +{ + SignatureEntry *entry; + Signature *vsig; + + map::const_iterator iter; + for(iter=sigmap.begin();iter!=sigmap.end();++iter) { + entry = (*iter).second; + if (entry->isNotEmitted()) continue; + vsig = new VarnodeSignature(entry->getVarnode(),entry->getHash()); + addSignature(vsig); + } +} + +/// For each basic block, we scan for operations that represent the roots of expressions: +/// CALL, CALLIND, CALLOTHER, STORE, CBRANCH, BRANCHIND, and RETURN. These are taken in sequence, as overlapping pairs, +/// generating cross-expression features. If there are stand-alone COPYs in the basic block, these are combined into +/// a single feature that is invariant under reordering of the COPYs. Finally a feature is generated that contains +/// pure control-flow information about the basic block. +void GraphSigManager::collectBlockSigs(void) + +{ + map::const_iterator iter; + for(iter=blockmap.begin();iter!=blockmap.end();++iter) { + BlockSignatureEntry *entry = (*iter).second; + BlockBasic *bl = entry->getBlock(); + + PcodeOp *op,*lastop; + SignatureEntry *outEntry; + hashword lasthash; + hashword val,callhash,copyhash,finalhash; + + lastop = (PcodeOp *)0; + lasthash = 0; + callhash = 0; + copyhash = 0; + list::const_iterator oiter,enditer; + oiter = bl->beginOp(); + enditer = bl->endOp(); + while(oiter != enditer) { + op = *oiter; + ++oiter; + int4 startind = 0; + int4 stopind = 0; + switch(op->code()) { + case CPUI_CALL: + // We don't care if the output is used or not, always include + callhash += 100001; + callhash *= 0x78abbf; + startind = 1; + stopind = op->numInput(); + break; + case CPUI_CALLIND: + // We don't care if the output is used or not, always include + callhash += 123451; // Slightly different than CPUI_CALL + callhash *= 0x78abbf; + startind = 1; + stopind = op->numInput(); + break; + case CPUI_CALLOTHER: + // We don't care if the output is used or not, always include + startind = 1; + stopind = op->numInput(); + break; + case CPUI_STORE: + startind = 1; + stopind = op->numInput(); + break; + case CPUI_CBRANCH: + startind = 1; + stopind = 2; + break; + case CPUI_BRANCHIND: + startind = 0; + stopind = 1; + break; + case CPUI_RETURN: + startind = 1; + stopind = op->numInput(); + break; + case CPUI_INDIRECT: + case CPUI_COPY: + outEntry = SignatureEntry::mapToEntry(op->getOut(), sigmap); + if (outEntry->isStandaloneCopy()) { + copyhash += outEntry->getHash(); + } + continue; + default: + startind = 0; // Don't use + stopind = 0; + break; + } + Varnode *outvn = op->getOut(); + if ((stopind == 0) && (outvn == (Varnode *)0 || !outvn->hasNoDescend())) continue; + if (outvn != (Varnode *) 0) { + outEntry = SignatureEntry::mapToEntry(outvn, sigmap); + if (outEntry->isNotEmitted()) continue; + val = outEntry->getHash(); + } + else { + val = (hashword) op->code(); + val = val ^ (val << 9) ^ (val << 18); + hashword accum = 0; + for (int4 i = startind; i < stopind; ++i) { // Let hash be invariant under commutivity + Varnode *vn = op->getIn(i); + hashword tmphash = hash_mixin(val, SignatureEntry::mapToEntryCollapse(vn, sigmap)->getHash()); + accum += tmphash; + } + val ^= accum; // Even if no-inputs we still get hash of opcode + } + if (lastop == (PcodeOp *) 0) + finalhash = hash_mixin(val, entry->getHash()); + else + finalhash = hash_mixin(val, lasthash); + Signature *bsig = new BlockSignature(bl, finalhash, lastop, op); + addSignature(bsig); + lastop = op; + lasthash = val; + } + finalhash = hash_mixin(entry->getHash(),0x9b1c5f); // Create a hash with just block information + if (callhash != 0) + finalhash = hash_mixin(finalhash,callhash); + addSignature(new BlockSignature(bl,finalhash,(PcodeOp *)0,(PcodeOp *)0)); + if (copyhash != 0) { + copyhash = hash_mixin(copyhash, 0xa2de3c); + addSignature(new CopySignature(bl,copyhash)); + } + } +} + +void GraphSigManager::varnodeClear(void) + +{ + map::iterator iter; + + for(iter=sigmap.begin();iter!=sigmap.end();++iter) + delete (*iter).second; + + sigmap.clear(); +} + +void GraphSigManager::blockClear(void) + +{ + map::iterator iter; + + for(iter=blockmap.begin();iter!=blockmap.end();++iter) + delete (*iter).second; + blockmap.clear(); +} + +/// Every basic block in the current function is allocated a BlockSignatureEntry and +/// local hash information is calculation in preparation for iterating. +void GraphSigManager::initializeBlocks(void) + +{ + const BlockGraph &blockgraph(fd->getBasicBlocks()); + for(int4 i=0;igetIndex() ] = entry; + entry->localHash(sigmods); + } +} + +/// \return \b true if the settings are valid for \b this manager +bool GraphSigManager::testSettings(uint4 val) + +{ + if (val == 0) + return false; // 0 setting is not allowed + // Allowed setting bits + uint4 mask = SIG_COLLAPSE_SIZE | SIG_DONOTUSE_CONST | SIG_DONOTUSE_INPUT | + SIG_DONOTUSE_PERSIST | SIG_COLLAPSE_INDNOISE; + mask = (mask << 2) | 1; // Add the check bit + return ((val & ~mask) == 0); // Do not allow any other bit to be set +} + +GraphSigManager::GraphSigManager(void) : SigManager() + +{ + // Set reasonable defaults + uint4 setting = SigManager::getSettings(); + if (!testSettings(setting)) + throw LowlevelError("Bad signature settings"); + sigmods = setting >> 2; + maxiter = 3; + maxblockiter = 1; + maxvarnode = 0; +} + +void GraphSigManager::clear(void) + +{ + varnodeClear(); + blockClear(); + SigManager::clear(); +} + +void GraphSigManager::initializeFromStream(istream &s) + +{ + int4 mymaxiter; + + s.unsetf(ios::dec | ios::hex | ios::oct); // Let user specify base + mymaxiter = -1; + s >> ws >> mymaxiter; + + if (mymaxiter!=-1) + maxiter = mymaxiter; +} + +void GraphSigManager::setCurrentFunction(const Funcdata *f) + +{ + SigManager::setCurrentFunction(f); + + VarnodeLocSet::const_iterator iter; + int4 size = f->numVarnodes(); + if ((maxvarnode!=0)&&(size > maxvarnode)) + throw LowlevelError(f->getName() + " exceeds size threshold for generating signatures"); + + for(iter=f->beginLoc();iter!=f->endLoc();++iter) { + Varnode *vn = *iter; + SignatureEntry *entry = new SignatureEntry(vn,sigmods); + sigmap[ vn->getCreateIndex() ] = entry; + } + map::const_iterator sigiter; + if ((sigmods & SIG_COLLAPSE_INDNOISE)!=0) { + SignatureEntry::removeNoise(sigmap); +#ifdef COPYNOISE_DEBUG + SignatureEntry::verifyAllNoiseRemoval(sigmap); +#endif + } + else { + for(sigiter=sigmap.begin();sigiter!=sigmap.end();++sigiter) + (*sigiter).second->calculateShadow(sigmap); + } + for(sigiter=sigmap.begin();sigiter!=sigmap.end();++sigiter) { + SignatureEntry *entry = (*sigiter).second; + entry->localHash(sigmods); + } +} + +void GraphSigManager::flipVarnodes(void) + +{ + map::iterator iter; + + for(iter=sigmap.begin();iter!=sigmap.end();++iter) { + SignatureEntry *entry = (*iter).second; + entry->flip(); + } +} + +void GraphSigManager::flipBlocks(void) + +{ + map::iterator iter; + + for(iter=blockmap.begin();iter!=blockmap.end();++iter) { + BlockSignatureEntry *entry = (*iter).second; + entry->flip(); + } +} + +/// Run through every Varnode (via its SignatureEntry overlay) and combine its current hash value +/// with the current hash value of the Varnode inputs to its effective defining PcodeOp. +void GraphSigManager::signatureIterate(void) + +{ + int4 j; + SignatureEntry *entry,*vnentry; + vector neigh; + map::const_iterator iter; + + flipVarnodes(); + for(iter=sigmap.begin();iter!=sigmap.end();++iter) { + entry = (*iter).second; + if (entry->isNotEmitted()) continue; + if (entry->isTerminal()) continue; + int4 num = entry->numInputs(); + neigh.clear(); + for(j=0;jgetIn(j,sigmap); + neigh.push_back(vnentry); + } + entry->hashIn(neigh); + } +} + +/// Run through every basic block (via its BlockSignatureEntry overlay) and combine its current hash value +/// with the current hash value of the incoming basic blocks. +void GraphSigManager::signatureBlockIterate(void) + +{ + vector neigh; + + flipBlocks(); + map::const_iterator iter,biter; + + for(iter=blockmap.begin();iter!=blockmap.end();++iter) { + BlockSignatureEntry *entry = (*iter).second; + BlockBasic *bl = entry->getBlock(); + neigh.clear(); + for(int4 i=0;isizeIn();++i) { + FlowBlock *inbl = bl->getIn(i); + biter = blockmap.find(inbl->getIndex()); + BlockSignatureEntry *inentry = (*biter).second; + neigh.push_back(inentry); + } + entry->hashIn(neigh); + } +} + +void GraphSigManager::generate(void) + +{ + int4 minusone,firsthalf,secondhalf; + + minusone = maxiter - 1; + firsthalf = minusone/2; + secondhalf = minusone - firsthalf; + signatureIterate(); + for(int4 i=0;i=0 ) { + initializeBlocks(); + for(int4 i=0;i feature; + sigmanager.getSignatureVector(feature); + encoder.openElement(ELEM_SIGNATURES); + if (fd->hasUnimplemented()) + encoder.writeBool(ATTRIB_UNIMPL, true); + if (fd->hasBadData()) + encoder.writeBool(ATTRIB_BADDATA, true); + for(uint4 i=0;inumCalls(); + for(uint4 i=0;igetCallSpecs(i); + const Address &addr(fc->getEntryAddress()); + if (!addr.isInvalid()) { + encoder.openElement(ELEM_CALL); + encoder.writeSpace(ATTRIB_SPACE, addr.getSpace()); + encoder.writeUnsignedInteger(ATTRIB_OFFSET, addr.getOffset()); + encoder.closeElement(ELEM_CALL); + } + } + encoder.closeElement(ELEM_SIGNATURES); +} + +/// Features are generated for the function and a complete description of each feature is +/// written to the encoder. The function must have been previously decompiled. +/// \param fd is the function to extract features from +/// \param encoder is the stream encoder to write output to +void debugSignature(Funcdata *fd,Encoder &encoder) + +{ + GraphSigManager sigmanager; + + sigmanager.setCurrentFunction(fd); + sigmanager.generate(); + sigmanager.sortByHash(); + sigmanager.encode(encoder); +} + +} // End namespace ghidra diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/signature.hh b/Ghidra/Features/Decompiler/src/decompile/cpp/signature.hh new file mode 100755 index 0000000000..3d19697175 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/signature.hh @@ -0,0 +1,357 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +/// \file signature.hh +/// \brief Classes for generating feature vectors representing individual functions +#ifndef __SIGNATURE_HH__ +#define __SIGNATURE_HH__ + +#include "funcdata.hh" + +namespace ghidra { + +typedef uint8 hashword; ///< Data-type for containing hash information + +extern AttributeId ATTRIB_BADDATA; ///< Marshaling attribute "baddata" +extern AttributeId ATTRIB_HASH; ///< Marshaling attribute "hash" +extern AttributeId ATTRIB_UNIMPL; ///< Marshaling attribute "unimpl" + +extern ElementId ELEM_BLOCKSIG; ///< Marshaling element \ +extern ElementId ELEM_CALL; ///< Marshaling element \ +extern ElementId ELEM_GENSIG; ///< Marshaling element \ +extern ElementId ELEM_MAJOR; ///< Marshaling element \ +extern ElementId ELEM_MINOR; ///< Marshaling element \ +extern ElementId ELEM_COPYSIG; ///< Marshaling element \ +extern ElementId ELEM_SETTINGS; ///< Marshaling element \ +extern ElementId ELEM_SIG; ///< Marshaling element \ +extern ElementId ELEM_SIGNATUREDESC; ///< Marshaling element \ +extern ElementId ELEM_SIGNATURES; ///< Marshaling element \ +extern ElementId ELEM_SIGSETTINGS; ///< Marshaling element \ +extern ElementId ELEM_VARSIG; ///< Marshaling element \ + +/// \brief A \b feature describing some aspect of a function or other unit of code +/// +/// The underlying representation is just a 32-bit hash of the \e information representing +/// the feature, but derived classes may be contain other meta-data describing where and how the +/// feature was formed. Two features are generally unordered (they are either equal or not equal), +/// but an ordering is used internally to normalize the vector representation and accelerate comparison. +class Signature { + uint4 sig; ///< Underlying 32-bit hash +public: + Signature(hashword h) { sig=(uint4)h; } ///< Constructor + uint4 getHash(void) const { return sig; } ///< Get the underyling 32-bit hash of the feature + void print(ostream &s) const; ///< Print the feature hash and a brief description of \b this feature to the given stream + int4 compare(const Signature *op2) const; ///< Compare two features + virtual ~Signature(void) {} ///< Destructor + virtual void encode(Encoder &encoder) const; /// Encode \b this feature to the given stream + virtual void decode(Decoder &decoder); /// Restore \b this feature from the given stream + + /// \brief Print a brief description of \b this feature to the given stream + virtual void printOrigin(ostream &s) const { + s << hex << "0x" << setfill('0') << setw(8) << sig; + } + + /// \brief Compare two Signature pointers via their underlying hash values + static bool comparePtr(Signature *a,Signature *b) { return (a->sig < b->sig); } +}; + +/// \brief A node for data-flow \b feature generation +/// +/// A SignatureEntry is rooted at a specific Varnode in the data-flow of a function. +/// During feature generation it iteratively hashes information about the Varnode and its nearest +/// neighbors through the edges of the graph. Feature generation needs to explicitly label: +/// - Varnodes that don't contribute meaningful information +/// - Shadow Varnodes that are direct or indirect COPYs of other Varnodes +/// - Stand-alone COPYs from a constant or input to a Varnode that is not directly read from again +class SignatureEntry { + /// Varnode properties that need to be explicit during feature generation + enum SignatureFlags { + SIG_NODE_TERMINAL = 0x1, ///< Varnode has no incoming edges + SIG_NODE_COMMUTATIVE = 0x2, ///< No distinction between this Varnode's incoming edges + SIG_NODE_NOT_EMITTED = 0x4, ///< Varnode is not emitted as a formal feature (it might be hashed with other features) + SIG_NODE_STANDALONE = 0x8, ///< Varnode is a stand-alone COPY + VISITED = 0x10, ///< Mark for spanning tree construction + MARKER_ROOT = 0x20 ///< Special root status in marker subgraph + }; + /// \brief A path node for doing depth first traversals of data-flow informed by SignatureEntry + struct DFSNode { + SignatureEntry *entry; ///< The specific node in the traversal path + list::const_iterator iter; ///< The edge to the next node in the path + }; + Varnode *vn; ///< The root Varnode + uint4 flags; ///< Feature generation properties of this Varnode + hashword hash[2]; ///< Current and previous hash + const PcodeOp *op; ///< The \e effective defining PcodeOp of this Varnode + int4 startvn; ///< First incoming edge (via the \e effective PcodeOp) + int4 inSize; ///< Number of incoming edges + int4 index; ///< Post-order index + SignatureEntry *shadow; ///< (If non-null) the Varnode being \e shadowed by this + hashword getOpHash(uint4 modifiers); ///< Get a hash encoding the OpCode of the \e effective defining PcodeOp + bool isVisited(void) const { return ((flags&VISITED)!=0); } ///< Return \b true if \b this node has been visited before + void setVisited(void) { flags |= VISITED; } ///< Mark that \b this node has been visited + + /// \brief Get the number of input edges for \b this in the noise reduced form of the data-flow graph + /// + /// \return the number of input edges + int4 markerSizeIn(void) const { + if ((flags&MARKER_ROOT)!=0) return 1; + return numInputs(); + } + + /// \brief Get a specific node coming into \b this in the noise reduced form of the data-flow graph + /// + /// \param i is the index of the incoming node + /// \param vRoot is the virtual root of the noise reduced form + /// \param sigMap is the map from a Varnode to its SignatureEntry overlay + /// \return the incoming SignatureEntry + SignatureEntry *getMarkerIn(int4 i,SignatureEntry *vRoot,const map &sigMap) const { + if ((flags&MARKER_ROOT)!=0) return vRoot; + return mapToEntry(op->getIn(i+startvn),sigMap); + } + + void standaloneCopyHash(uint4 modifiers); ///< Calculate the hash for stand-alone COPY + static bool testStandaloneCopy(Varnode *vn); ///< Determine if the given Varnode is a stand-alone COPY + static void noisePostOrder(const vector &rootlist,vector &postOrder,map &sigMap); + static void noiseDominator(vector &postOrder,map &sigMap); +public: + SignatureEntry(Varnode *v,uint4 modifiers); ///< Construct from a Varnode + SignatureEntry(int4 ind); ///< Construct a virtual node + bool isTerminal(void) const { return ((flags&SIG_NODE_TERMINAL)!=0); } ///< Return \b true if \b this node has no inputs + bool isNotEmitted(void) const { return ((flags&SIG_NODE_NOT_EMITTED)!=0); } ///< Return \b true if \b this is not emitted as a feature + bool isCommutative(void) const { return ((flags&SIG_NODE_COMMUTATIVE)!=0); } ///< Return \b true if inputs to \b this are unordered + bool isStandaloneCopy(void) const { return ((flags&SIG_NODE_STANDALONE)!=0); } ///< Return \b true if \b this is a stand-alone COPY + int4 numInputs(void) const { return inSize; } ///< Return the number incoming edges to \b this node + + /// \brief Get the i-th incoming node + /// + /// \param i is the index + /// \param sigMap is the map from Varnode to its SignatureEntry overlay + /// \return the selected incoming SignatureEntry node + SignatureEntry *getIn(int4 i,const map &sigMap) const { + return mapToEntryCollapse(op->getIn(i+startvn),sigMap); + } + + void calculateShadow(const map &sigMap); ///< Determine if \b this node shadows another + void localHash(uint4 modifiers); ///< Compute an initial hash based on local properties of the Varnode + void flip(void) { hash[1] = hash[0]; } ///< Store hash from previous iteration and prepare for next iteration + void hashIn(vector &neigh); ///< Hash info from other nodes into \b this + Varnode *getVarnode(void) const { return vn; } ///< Get the underlying Varnode which \b this overlays + hashword getHash(void) const { return hash[0]; } ///< Get the current hash value + static SignatureEntry *mapToEntry(const Varnode *vn,const map &sigMap); + static SignatureEntry *mapToEntryCollapse(const Varnode *vn,const map &sigMap); + static void removeNoise(map &sigMap); + static hashword hashSize(Varnode *vn,uint4 modifiers); +#ifdef COPYNOISE_DEBUG + void verifyNoiseRemoval(map &sigMap) const; ///< Verify \b shadow is set correctly for \b this + static void verifyAllNoiseRemoval(map &sigMap); ///< Verify all nodes have \b shadow set correctly +#endif +}; + +/// \brief A node for control-flow feature generation +/// +/// A BlockSignatureEntry is rooted at a specific basic block in the control-flow of a function. +/// During feature generation it iteratively hashes information about the basic block and its +/// nearest neighbors through the edges of the control-flow graph. +class BlockSignatureEntry { + BlockBasic *bl; ///< The root basic block + hashword hash[2]; ///< Current and previous hash +public: + BlockSignatureEntry(BlockBasic *b) { bl = b; } ///< Construct from a basic block + void localHash(uint4 modifiers); ///< Compute an initial hash based on local properties of the basic block + void flip(void) { hash[1] = hash[0]; } ///< Store hash from previous iteration and prepare for next iteration + void hashIn(vector &neigh); ///< Hash info from other nodes into \b this + BlockBasic *getBlock(void) const { return bl; } ///< Get the underlying basic block which \b this overlays + hashword getHash(void) const { return hash[0]; } ///< Get the current hash value +}; + +/// \brief A \e feature representing a portion of the data-flow graph rooted at a particular Varnode +/// +/// The feature recursively incorporates details about the Varnode, the PcodeOp that defined it and +/// its input Varnodes, up to a specific depth. +class VarnodeSignature : public Signature { + const Varnode *vn; ///< The root Varnode +public: + VarnodeSignature(const Varnode *v,hashword h) : Signature(h) { vn = v; } ///< Constructor + virtual void encode(Encoder &encoder) const; + virtual void printOrigin(ostream &s) const { vn->printRaw(s); } +}; + +/// \brief A \e feature rooted in a basic block +/// +/// There are two forms of a block feature. +/// Form 1 contains only local control-flow information about the basic block. +/// Form 2 is a feature that combines two operations that occur in sequence within the block. +/// This form incorporates info about the operations and data-flow info about their inputs. +class BlockSignature : public Signature { + const BlockBasic *bl; ///< The root basic block + const PcodeOp *op1; ///< (Form 2)The first operation in sequence in the feature + const PcodeOp *op2; ///< (Form 2)The second operation in sequence in the feature +public: + BlockSignature(const BlockBasic *b,hashword h, + const PcodeOp *o1,const PcodeOp *o2) : Signature(h) + { bl = b; op1 = o1; op2 = o2; } ///< Constructor + virtual void encode(Encoder &encoder) const; + virtual void printOrigin(ostream &s) const { bl->printHeader(s); } +}; + +/// \brief A feature representing 1 or more \e stand-alone copies in a basic block +/// +/// A COPY operation is considered stand-alone if either a constant or a function input +/// is copied into a location that is then not read directly by the function. +/// These COPYs are incorporated into a single feature, which encodes the number +/// and type of COPYs but does not encode the order in which they occur within the block. +class CopySignature : public Signature { + const BlockBasic *bl; ///< The basic block containing the COPY +public: + CopySignature(const BlockBasic *b,hashword h) + : Signature(h) { bl = b; } ///< Constructor + virtual void encode(Encoder &encoder) const; + virtual void printOrigin(ostream &s) const; +}; + +/// \brief A container for collecting a set of features (a feature vector) for a single function +/// +/// This manager handles: +/// - Configuring details of the signature generation process +/// - Establishing the function being signatured , via setCurrentFunction() +/// - Generating the features, via generate() +/// - Outputting the features, via encode() or print() +/// +/// The manager can be reused for multiple functions. +class SigManager { + static uint4 settings; ///< Signature settings (across all managers) + vector sigs; ///< Feature set for the current function + void clearSignatures(void); ///< Clear all current Signature/feature objects from \b this manager +protected: + const Funcdata *fd; ///< Current function off of which we are generating features + void addSignature(Signature *sig) { sigs.push_back(sig); } ///< Add a new feature to the manager +public: + SigManager(void) { fd = (const Funcdata *)0; } ///< Constructor + virtual ~SigManager(void) { clearSignatures(); } ///< Destructor + virtual void clear(void); ///< Clear all current Signature/feature resources + virtual void initializeFromStream(istream &s)=0; ///< Read configuration information from a character stream + virtual void setCurrentFunction(const Funcdata *f); ///< Set the function used for (future) feature generation + virtual void generate(void)=0; ///< Generate all features for the current function + int4 numSignatures(void) const { return sigs.size(); } ///< Get the number of features currently generated + Signature *getSignature(int4 i) const { return sigs[i]; } ///< Get the i-th Signature/feature + void getSignatureVector(vector &feature) const; ///< Get the feature vector as a simple array of hashes + hashword getOverallHash(void) const; ///< Combine all feature hashes into one overall hash + void sortByHash(void) { sort(sigs.begin(),sigs.end(),Signature::comparePtr); } ///< Sort all current features + void print(ostream &s) const; ///< Print a brief description of all current features to a stream + void encode(Encoder &encoder) const; ///< Encode all current features to the given stream + static uint4 getSettings(void) { return settings; } ///< Get the settings currently being used for signature generation + static void setSettings(uint4 newvalue); ///< Establish settings to use for future signature generation +}; + +/// \brief A manager for generating Signatures/features on function data-flow and control-flow +/// +/// Features are extracted from the data-flow and control-flow graphs of the function. +/// The different feature types produced by this manager are: +/// - VarnodeSignature +/// - BlockSignature +/// - CopySignature +class GraphSigManager : public SigManager { +public: + /// Signature generation settings + enum Mods { + SIG_COLLAPSE_SIZE = 0x1, ///< Treat certain varnode sizes as the same + SIG_COLLAPSE_INDNOISE = 0x2, ///< Collapse varnodes that indirect copies of each other +// SIG_CALL_TERMINAL = 0x8, ///< Do not consider data-flow across CALLs + SIG_DONOTUSE_CONST = 0x10, ///< Do not use value of constant in hash + SIG_DONOTUSE_INPUT = 0x20, ///< Do not use (fact of) being an input in hash + SIG_DONOTUSE_PERSIST = 0x40 ///< Do not use (fact of) being a global in hash + }; +private: + uint4 sigmods; ///< Current settings to use for signature generation + int4 maxiter; ///< Maximum number of iterations across data-flow graph + int4 maxblockiter; ///< Maximum number of block iterations + int4 maxvarnode; ///< Maximum number of Varnodes to signature + map sigmap; ///< Map from Varnode to SignatureEntry overlay + map blockmap; ///< Map from basic block to BlockSignatureEntry overlay + void signatureIterate(void); ///< Do one iteration of hashing on the SignatureEntrys + void signatureBlockIterate(void); ///< Do one iteration of hashing on the BlockSignatureEntrys + void collectVarnodeSigs(void); ///< Generate the final feature for each Varnode from its SignatureEntry overlay + void collectBlockSigs(void); ///< Generate the final feature(s) for each basic block from its BlockSignatureEntry overlay + void varnodeClear(void); ///< Clear all SignatureEntry overlay objects + void blockClear(void); ///< Clear all BlockSignatureEntry overlay objects + void initializeBlocks(void); ///< Initialize BlockSignatureEntry overlays for the current function + void flipVarnodes(void); ///< Store off \e current Varnode hash values as \e previous hash values + void flipBlocks(void); ///< Store off \e current block hash values as \e previous hash values +public: + virtual void clear(void); + GraphSigManager(void); ///< Constructor + virtual ~GraphSigManager(void) { varnodeClear(); } ///< Destructor + void setMaxIteration(int4 val) { maxiter = val; } ///< Override the default iterations used for Varnode features + void setMaxBlockIteration(int4 val) { maxblockiter = val; } ///< Override the default iterations used for block features + void setMaxVarnode(int4 val) { maxvarnode = val; } ///< Set a maximum threshold for Varnodes in a function + virtual void initializeFromStream(istream &s); + virtual void setCurrentFunction(const Funcdata *f); + virtual void generate(void); + static bool testSettings(uint4 val); ///< Test for valid signature generation settings +}; + +/// \brief Given a Varnode, find its SignatureEntry overlay +/// +/// \param vn is the given Varnode +/// \param sigMap is the map from Varnode to SignatureEntry +/// \return the corresponding SignatureEntry +inline SignatureEntry *SignatureEntry::mapToEntry(const Varnode *vn,const map &sigMap) + +{ + map::const_iterator iter; + + iter = sigMap.find(vn->getCreateIndex()); + return (*iter).second; +} + +/// \brief Given a Varnode, find its SignatureEntry overlay, collapsing shadows +/// +/// If the corresponding SignatureEntry shadows another, the shadowed SignatureEntry is returned instead. +/// \param vn is the given Varnode +/// \param sigMap is the map from Varnode to SignatureEntry +/// \return the corresponding SignatureEntry +inline SignatureEntry *SignatureEntry::mapToEntryCollapse(const Varnode *vn,const map &sigMap) + +{ + SignatureEntry *res = mapToEntry(vn,sigMap); + if (res->shadow == (SignatureEntry *)0) + return res; + return res->shadow; +} + +/// \brief Calculate a hash describing the size of a given Varnode +/// +/// The hash is computed from the size of the Varnode in bytes, as an integer value. +/// Depending on the signature settings, the hash incorporates the full value, or +/// it may truncate a value greater than 4. +/// \param vn is the given Varnode +/// \param modifiers are the settings being used for signature generation +/// \return the hash value +inline hashword SignatureEntry::hashSize(Varnode *vn,uint4 modifiers) + +{ + hashword val = (hashword) vn->getSize(); // Size of varnode + if ((modifiers&GraphSigManager::SIG_COLLAPSE_SIZE)!=0) { + if (val>4) // Treat sizes 4 and larger the same + val = 4; + } + return val ^ (val<<7) ^ (val<<14) ^ (val<<21); +} + +extern void simpleSignature(Funcdata *fd,Encoder &encoder); ///< Generate features for a single function +extern void debugSignature(Funcdata *fd,Encoder &encoder); ///< Generate features (with debug info) for a single function + +} // End namespace ghidra +#endif diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/signature_ghidra.cc b/Ghidra/Features/Decompiler/src/decompile/cpp/signature_ghidra.cc new file mode 100755 index 0000000000..479ef00114 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/signature_ghidra.cc @@ -0,0 +1,119 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +#include "signature_ghidra.hh" +#include "signature.hh" + +namespace ghidra { + +// Constructing the singleton registers the capability +GhidraSignatureCapability GhidraSignatureCapability::ghidraSignatureCapability; + +void GhidraSignatureCapability::initialize(void) + +{ + commandmap["generateSignatures"] = new SignaturesAt(false); + commandmap["debugSignatures"] = new SignaturesAt(true); + commandmap["getSignatureSettings"] = new GetSignatureSettings(); + commandmap["setSignatureSettings"] = new SetSignatureSettings(); +} + +void SignaturesAt::loadParameters(void) + +{ + GhidraCommand::loadParameters(); + PackedDecode decoder(ghidra); + ArchitectureGhidra::readStringStream(sin,decoder); + addr = Address::decode(decoder); // Parse XML for functions address +} + +void SignaturesAt::rawAction(void) + +{ + Funcdata *fd = ghidra->symboltab->getGlobalScope()->queryFunction(addr); + if (fd == (Funcdata *)0) { + ostringstream s; + s << "Bad address for signatures: " << addr.getShortcut(); + addr.printRaw(s); + s << '\n'; + throw LowlevelError(s.str()); + } + if (!fd->isProcStarted()) { + string curname = ghidra->allacts.getCurrentName(); + Action *sigact; + if (curname != "normalize") + sigact = ghidra->allacts.setCurrent("normalize"); + else + sigact = ghidra->allacts.getCurrent(); +#ifdef __REMOTE_SOCKET__ + connect_to_console(fd); +#endif + sigact->reset(*fd); + sigact->perform(*fd); + if (curname != "normalize") + ghidra->allacts.setCurrent(curname); + } + + sout.write("\000\000\001\016",4); + PackedEncode encoder(sout); // Write output XML directly to outstream + if (debug) + debugSignature(fd,encoder); + else + simpleSignature(fd,encoder); + sout.write("\000\000\001\017",4); +} + +void GetSignatureSettings::rawAction(void) + +{ + sout.write("\000\000\001\016",4); // Write output XML directly to outstream + PackedEncode encoder(sout); + encoder.openElement(ELEM_SIGSETTINGS); + encoder.openElement(ELEM_MAJOR); + encoder.writeSignedInteger(ATTRIB_CONTENT, ArchitectureCapability::getMajorVersion()); + encoder.closeElement(ELEM_MAJOR); + encoder.openElement(ELEM_MINOR); + encoder.writeSignedInteger(ATTRIB_CONTENT, ArchitectureCapability::getMinorVersion()); + encoder.closeElement(ELEM_MINOR); + encoder.openElement(ELEM_SETTINGS); + encoder.writeUnsignedInteger(ATTRIB_CONTENT, SigManager::getSettings()); + encoder.closeElement(ELEM_SETTINGS); + encoder.closeElement(ELEM_SIGSETTINGS); + sout.write("\000\000\001\017",4); +} + +void SetSignatureSettings::loadParameters(void) + +{ + string settingString; + GhidraCommand::loadParameters(); + ArchitectureGhidra::readStringStream(sin,settingString); + istringstream s(settingString); + s.unsetf(ios::dec | ios::hex | ios::oct); + s >> settings; +} + +void SetSignatureSettings::rawAction(void) + +{ + if (GraphSigManager::testSettings(settings)) { + SigManager::setSettings(settings); + ArchitectureGhidra::writeStringStream(sout,"t"); + } + else + ArchitectureGhidra::writeStringStream(sout,"f"); +} + +} // End namespace ghidra diff --git a/Ghidra/Features/Decompiler/src/decompile/cpp/signature_ghidra.hh b/Ghidra/Features/Decompiler/src/decompile/cpp/signature_ghidra.hh new file mode 100755 index 0000000000..c7a34aa29c --- /dev/null +++ b/Ghidra/Features/Decompiler/src/decompile/cpp/signature_ghidra.hh @@ -0,0 +1,78 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +/// \file signature_ghidra.hh +/// \brief Feature/Signature generation commands that can be issued to the decompiler by the Ghidra client +#ifndef __GHIDRA_SIGNATURES_HH__ +#define __GHIDRA_SIGNATURES_HH__ + +#include "ghidra_process.hh" + +namespace ghidra { + +/// \brief Signature command capability +/// +/// This class is instantiated as a singleton and registers commands that the Ghidra client can issue +/// for generating feature vectors extracted from decompiled functions. +class GhidraSignatureCapability : public GhidraCapability { + static GhidraSignatureCapability ghidraSignatureCapability; ///< Singleton instance + GhidraSignatureCapability(void) { name = "signature"; } ///< Construct the singleton instance + GhidraSignatureCapability(const GhidraSignatureCapability &op2); ///< Not implemented + GhidraSignatureCapability &operator=(const GhidraSignatureCapability &op2); ///< Not implemented +public: + virtual void initialize(void); +}; + +/// \brief Command to generate a feature vector from a function's data-flow and control-flow graphs +/// +/// The command expects to receive the entry point address of a function. The function is +/// decompiled using the "normalize" simplification style. Then features are extracted from the +/// resulting data-flow and control-flow graphs of the decompiled function. The features are +/// returned to the Ghidra client. The command can be instantiated in two forms. One form returns +/// a stream-lined encoding of the feature vector for more efficient transfers during normal operation. +/// The other form returns more descriptive meta-data with the features and is suitable for debugging +/// or exploring the feature generation process. +class SignaturesAt : public GhidraCommand { + bool debug; ///< True if the command should return verbose feature encodings + Address addr; ///< The entry point of the function to generate features for + virtual void loadParameters(void); +public: + SignaturesAt(bool dbg) { debug = dbg; } ///< Constructor specifying response format + virtual void rawAction(void); +}; + +/// \brief Command to retrieve current decompiler settings being used for feature/signature generation +/// +/// The command returns an opaque integer indicating the state of boolean properties affecting +/// feature generation. The reserved value of 0 indicates that no settings have been provided to the +/// decompiler process. +class GetSignatureSettings : public GhidraCommand { +public: + virtual void rawAction(void); +}; + +/// \brief Command to provide the global settings used by the decompiler process during feature/signature generation +/// +/// The command expects to receive an opaque integer value encoding the state of boolean properties affecting +/// feature generation. The command returns 't' indicating a valid setting was received or 'f' for an invalid setting. +class SetSignatureSettings : public GhidraCommand { + uint4 settings; ///< Opaque settings value being requested + virtual void loadParameters(void); +public: + virtual void rawAction(void); +}; + +} // End namespace ghidra +#endif diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/DecompInterface.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/DecompInterface.java index 021500fd93..3ab6647a51 100644 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/DecompInterface.java +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/DecompInterface.java @@ -13,17 +13,18 @@ * See the License for the specific language governing permissions and * limitations under the License. */ -/* - * Created on Oct 29, 2003 - * - * Deals with the direct interface between C/C++ decompiler - */ package ghidra.app.decompiler; +import static ghidra.program.model.pcode.AttributeId.*; +import static ghidra.program.model.pcode.ElementId.*; + import java.io.*; +import java.util.ArrayList; import generic.jar.ResourceFile; +import ghidra.app.decompiler.signature.DebugSignature; +import ghidra.app.decompiler.signature.SignatureResult; import ghidra.app.plugin.processors.sleigh.SleighLanguage; import ghidra.app.plugin.processors.sleigh.UniqueLayout; import ghidra.program.model.address.*; @@ -147,12 +148,15 @@ public class DecompInterface { }; // Initialization state - private String actionname; // Name of simplification action - private DecompileOptions options; // Current decompiler options - private boolean printSyntaxTree; // Whether syntax tree is returned - private boolean printCCode; // Whether C code is returned - private boolean sendParamMeasures; // Whether Parameter Measures are returned - private boolean jumpLoad; // Whether jumptable load information is returned + private String actionname; // Name of simplification action + private DecompileOptions options; // Current decompiler options + private boolean printSyntaxTree; // Whether syntax tree is returned + private boolean printCCode; // Whether C code is returned + private boolean sendParamMeasures; // Whether Parameter Measures are returned + private boolean jumpLoad; // Whether jumptable load information is returned + private short major; // Major decompiler version + private short minor; // Minor decompiler version + private int sigSettings; // Settings for signature generation (0=not configured) public DecompInterface() { program = null; @@ -170,6 +174,9 @@ public class DecompInterface { printCCode = true; sendParamMeasures = false; jumpLoad = false; + major = 0; // 0 indicates the major/minor values have yet to be fetched from decompiler + minor = 0; + sigSettings = 0; } /** @@ -330,6 +337,20 @@ public class DecompInterface { throw new IOException("Could not turn on jumptable loads"); } } + if (sigSettings != 0) { + decompProcess.sendCommand1Param("setSignatureSettings", Integer.toString(sigSettings), + stringResponse); + if (stringResponse.isEmpty()) { + if (decompCallback.getNativeMessage().startsWith("Bad command")) { + throw new DecompileException("Decompiler", + "Decompiler executable not built with signature module"); + } + throw new DecompileException("Decompiler", decompCallback.getNativeMessage()); + } + if (!stringResponse.toString().equals("t")) { + throw new IOException("Could not set signature settings"); + } + } } protected void verifyProcess() throws IOException, DecompileException { @@ -891,6 +912,217 @@ public class DecompInterface { overlayEncodingSet.setOverlay(overlay); } return overlayEncodingSet; + } + /** + * Read the major/minor version numbers from the stream. + * @param decoder is the stream decoder + * @throws DecoderException for problems parsing the stream or if the returned settings do not match + */ + private void decodeVersionNumber(Decoder decoder) throws DecoderException { + int el = decoder.openElement(ELEM_SIGSETTINGS); + int subel = decoder.openElement(ELEM_MAJOR); + major = (short) decoder.readSignedInteger(ATTRIB_CONTENT); + decoder.closeElement(subel); + subel = decoder.openElement(ELEM_MINOR); + minor = (short) decoder.readSignedInteger(ATTRIB_CONTENT); + decoder.closeElement(subel); + subel = decoder.openElement(ELEM_SETTINGS); + int settings = (int) decoder.readUnsignedInteger(ATTRIB_CONTENT); + decoder.closeElement(subel); + decoder.closeElement(el); + if (settings != sigSettings) { + throw new DecoderException("Decompiler settings are not configured"); + } + } + + /** + * Query the decompiler process for its major/minor version numbers. + * Additionally the decompiler returns its signature settings (which should match the + * signatureSettings configured on this interface) + */ + private void fillinVersionNumber() { + try { + verifyProcess(); + decompProcess.sendCommand("getSignatureSettings", baseEncodingSet.mainResponse); + decompileMessage = decompCallback.getNativeMessage(); + } + catch (Exception e) { + decompileMessage = "Exception while retrieving settings: " + e.getMessage() + '\n'; + } + // flushCache(); // We don't need to flush the cache + if (!baseEncodingSet.mainResponse.isEmpty()) { + try { + decodeVersionNumber(baseEncodingSet.mainResponse); + } + catch (Exception e) { + decompileMessage = "Exception while parsing signatures: " + e.getMessage() + '\n'; + } + } + } + + /** + * @return the major version number of the decompiler + */ + public synchronized short getMajorVersion() { + if (major == 0) { + fillinVersionNumber(); + } + return major; + } + + /** + * @return the minor version number of the decompiler + */ + public synchronized short getMinorVersion() { + if (major == 0) { + fillinVersionNumber(); + } + return minor; + } + + /** + * @return the signature settings of the decompiler + */ + public synchronized int getSignatureSettings() { + if (major == 0) { + fillinVersionNumber(); // Verify the process settings match the configured settings + } + return sigSettings; + } + + /** + * Set the desired signature generation settings. + * @param value is the new desired setting + * @return true if the settings took effect + */ + public boolean setSignatureSettings(int value) { + sigSettings = value; + // Property can be set before process exists + if (decompProcess == null) { + return true; + } + try { + verifyProcess(); + decompProcess.sendCommand1Param("setSignatureSettings", Integer.toString(sigSettings), + stringResponse); + return stringResponse.toString().equals("t"); + } + catch (IOException e) { + // don't care + } + catch (DecompileException e) { + // don't care + } + stopProcess(); + return false; + } + + /** + * Generate a signature, using the current signature settings, for the given function. + * The signature is returned as a raw feature vector, {@link SignatureResult}. + * @param func is the given function + * @param keepcalllist is true if direct call addresses are collected as part of the result + * @param timeoutSecs is the maximum amount of time to spend decompiling the function + * @param monitor is the TaskMonitor + * @return the feature vector + */ + public synchronized SignatureResult generateSignatures(Function func, boolean keepcalllist, + int timeoutSecs, TaskMonitor monitor) { + + decompileMessage = ""; + if (monitor != null && monitor.isCancelled()) { + return null; + } + + if (monitor != null) { + monitor.addCancelledListener(monitorListener); + } + Decoder decoder = null; + try { + Address funcEntry = func.getEntryPoint(); + decompCallback.setFunction(func, funcEntry, null); + verifyProcess(); + EncodeDecodeSet activeSet = setupEncodeDecode(funcEntry); + decoder = activeSet.mainResponse; + activeSet.mainQuery.clear(); + AddressXML.encode(activeSet.mainQuery, funcEntry); + decompProcess.sendCommandTimeout("generateSignatures", timeoutSecs, activeSet); + decompileMessage = decompCallback.getNativeMessage(); + } + catch (Exception ex) { + decompileMessage = "Exception while generating signatures: " + ex.getMessage() + '\n'; + } + finally { + if (monitor != null) { + monitor.removeCancelledListener(monitorListener); + } + } + flushCache(); + if (!decoder.isEmpty()) { + try { + return SignatureResult.decode(decoder, func, keepcalllist); + } + catch (DecoderException e) { // Error walking the DOM + decompileMessage = "Exception while parsing signatures: " + e.getMessage() + '\n'; + } + } + return null; + } + + /** + * Generate a signature, using the current signature settings, for the given function. + * The signature is returned as a sequence of features (feature vector). Each feature + * is returned as a separate record with additional metadata describing the information + * incorporated into it. + * @param func is the given function + * @param timeoutSecs is the maximum number of seconds to spend decompiling the function + * @param monitor is the TaskMonitor + * @return the array of feature descriptions + */ + public synchronized ArrayList debugSignatures(Function func, int timeoutSecs, + TaskMonitor monitor) { + decompileMessage = ""; + if (monitor != null && monitor.isCancelled()) { + return null; + } + + if (monitor != null) { + monitor.addCancelledListener(monitorListener); + } + Decoder decoder = null; + try { + Address funcEntry = func.getEntryPoint(); + decompCallback.setFunction(func, funcEntry, null); + verifyProcess(); + EncodeDecodeSet activeSet = setupEncodeDecode(funcEntry); + decoder = activeSet.mainResponse; + activeSet.mainQuery.clear(); + AddressXML.encode(activeSet.mainQuery, funcEntry); + decompProcess.sendCommandTimeout("debugSignatures", timeoutSecs, activeSet); + decompileMessage = decompCallback.getNativeMessage(); + } + catch (Exception ex) { + decompileMessage = "Exception while debugging signatures: " + ex.getMessage() + '\n'; + } + finally { + if (monitor != null) { + monitor.removeCancelledListener(monitorListener); + } + } + flushCache(); + if (!decoder.isEmpty()) { + try { + return DebugSignature.decodeSignatures(decoder, func); + } + catch (DecoderException e) { + decompileMessage = "Exception while debugging signatures: " + e.getMessage() + '\n'; + } + catch (Exception e) { // Error with the underlying stream + decompileMessage = + "Error in stream describing signatures: " + e.getMessage() + '\n'; + } + } + return null; } } diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/ApplyFunctionSignatureAction.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/ApplyFunctionSignatureAction.java deleted file mode 100644 index 7360831b65..0000000000 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/ApplyFunctionSignatureAction.java +++ /dev/null @@ -1,55 +0,0 @@ -/* ### - * IP: GHIDRA - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package ghidra.app.decompiler.component; - -import docking.ActionContext; -import docking.widgets.fieldpanel.internal.FieldPanelCoordinator; -import ghidra.app.plugin.core.functioncompare.actions.AbstractApplyFunctionSignatureAction; -import ghidra.app.util.viewer.util.CodeComparisonPanel; - -/** - * Action that applies the signature of the function in the currently active side of a decompiler - * code comparison panel to the function in the other side of the panel. - */ -public class ApplyFunctionSignatureAction extends AbstractApplyFunctionSignatureAction { - - /** - * Constructor for the action that applies a function signature from one side of a dual - * decompiler panel to the other. - * @param owner the owner of this action. - */ - public ApplyFunctionSignatureAction(String owner) { - super(owner); - } - - @Override - public boolean isAddToPopup(ActionContext context) { - return (context instanceof DualDecompilerActionContext); - } - - @Override - public boolean isEnabledForContext(ActionContext context) { - if (context instanceof DualDecompilerActionContext) { - DualDecompilerActionContext compareContext = (DualDecompilerActionContext) context; - CodeComparisonPanel codeComparisonPanel = - compareContext.getCodeComparisonPanel(); - if (codeComparisonPanel instanceof DecompilerCodeComparisonPanel) { - return !hasReadOnlyNonFocusedSide(codeComparisonPanel); - } - } - return false; - } -} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/BasicDecompilerCodeComparisonPanel.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/BasicDecompilerCodeComparisonPanel.java deleted file mode 100644 index 595a6b09d6..0000000000 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/BasicDecompilerCodeComparisonPanel.java +++ /dev/null @@ -1,45 +0,0 @@ -/* ### - * IP: GHIDRA - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package ghidra.app.decompiler.component; - -import ghidra.framework.plugintool.PluginTool; - -/** - * Panel that displays two decompilers for comparison and synchronizes their scrolling - * using a basic coordinator. - */ -public class BasicDecompilerCodeComparisonPanel - extends DecompilerCodeComparisonPanel { - - /** - * Creates a default comparison panel with two decompilers. - * @param owner the owner of this panel - * @param tool the tool displaying this panel - */ - public BasicDecompilerCodeComparisonPanel(String owner, PluginTool tool) { - super(owner, tool); - } - - @Override - public Class> getPanelThisSupersedes() { - return null; // Doesn't supersede any other panel. - } - - @Override - protected BasicDecompilerFieldPanelCoordinator createFieldPanelCoordinator() { - return new BasicDecompilerFieldPanelCoordinator(this, isScrollingSynced()); - } -} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/BasicDecompilerFieldPanelCoordinator.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/BasicDecompilerFieldPanelCoordinator.java deleted file mode 100644 index 9627d2a2bd..0000000000 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/BasicDecompilerFieldPanelCoordinator.java +++ /dev/null @@ -1,86 +0,0 @@ -/* ### - * IP: GHIDRA - * - * Licensed under the Apache License, Version 2.0 (the "License"); - * you may not use this file except in compliance with the License. - * You may obtain a copy of the License at - * - * http://www.apache.org/licenses/LICENSE-2.0 - * - * Unless required by applicable law or agreed to in writing, software - * distributed under the License is distributed on an "AS IS" BASIS, - * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. - * See the License for the specific language governing permissions and - * limitations under the License. - */ -package ghidra.app.decompiler.component; - -import docking.widgets.fieldpanel.support.FieldLocation; -import ghidra.program.util.ProgramLocation; - -/** - * A basic coordinator that locks two decompiler panels together at the first line so that - * scrolling one side also scrolls the other. It also allows the cursor locations to track - * together based on the line number or to move independent of each other. - */ -public class BasicDecompilerFieldPanelCoordinator extends DualDecompilerFieldPanelCoordinator { - - private DecompilerCodeComparisonPanel dualDecompilerPanel; - private boolean syncLineLocation; - - /** - * Constructs a dual decompiler coordinator that scrolls the two panels together so that - * they are locked together at the first line. - * @param dualDecompilerPanel the dual decompiler panel being controlled by this coordinator - * @param syncLineLocation true means synchronize the cursors in the two decompiler panels - * to the same line number and offset if possible. false means the the cursors move - * independently of each other. - */ - public BasicDecompilerFieldPanelCoordinator( - BasicDecompilerCodeComparisonPanel dualDecompilerPanel, boolean syncLineLocation) { - super(dualDecompilerPanel); - this.dualDecompilerPanel = dualDecompilerPanel; - this.syncLineLocation = syncLineLocation; - } - - @Override - public void leftLocationChanged(ProgramLocation leftProgramLocation) { - - if (syncLineLocation) { - CDisplayPanel focusedDecompilerPanel = dualDecompilerPanel.getFocusedDecompilerPanel(); - - CDisplayPanel leftPanel = dualDecompilerPanel.getLeftPanel(); - CDisplayPanel rightPanel = dualDecompilerPanel.getRightPanel(); - if (focusedDecompilerPanel != leftPanel) { - return; // Don't respond to location change from synchronizing left and right. - } - - DecompilerPanel leftDecompilerPanel = leftPanel.getDecompilerPanel(); - DecompilerPanel rightDecompilerPanel = rightPanel.getDecompilerPanel(); - - FieldLocation leftFieldLocation = leftDecompilerPanel.getCursorPosition(); - rightDecompilerPanel.setCursorPosition(leftFieldLocation); - } - } - - @Override - public void rightLocationChanged(ProgramLocation rightProgramLocation) { - - if (syncLineLocation) { - CDisplayPanel focusedDecompilerPanel = dualDecompilerPanel.getFocusedDecompilerPanel(); - - CDisplayPanel leftPanel = dualDecompilerPanel.getLeftPanel(); - CDisplayPanel rightPanel = dualDecompilerPanel.getRightPanel(); - if (focusedDecompilerPanel != rightPanel) { - return; // Don't respond to location change from synchronizing left and right. - } - - DecompilerPanel leftDecompilerPanel = leftPanel.getDecompilerPanel(); - DecompilerPanel rightDecompilerPanel = rightPanel.getDecompilerPanel(); - - FieldLocation rightFieldLocation = rightDecompilerPanel.getCursorPosition(); - leftDecompilerPanel.setCursorPosition(rightFieldLocation); - } - } - -} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/CDisplayPanel.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/CDisplayPanel.java index dd4a231bb3..ecb389addb 100644 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/CDisplayPanel.java +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/CDisplayPanel.java @@ -175,4 +175,13 @@ public class CDisplayPanel extends JPanel implements DecompilerCallbackHandler { public void dispose() { controller.dispose(); } + + public DecompilerController getController() { + return controller; + + } + + public void refresh(DecompileData data) { + controller.refreshDisplay(data.getProgram(), data.getLocation(), null); + } } diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DecompilerCodeComparisonPanel.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DecompilerCodeComparisonPanel.java index 5f636ab9f6..319a7f5735 100644 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DecompilerCodeComparisonPanel.java +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DecompilerCodeComparisonPanel.java @@ -17,13 +17,14 @@ package ghidra.app.decompiler.component; import java.awt.*; import java.awt.event.*; -import java.util.ArrayList; +import java.util.*; import javax.swing.*; import docking.ActionContext; import docking.ComponentProvider; import docking.action.*; +import docking.options.OptionsService; import docking.widgets.fieldpanel.FieldPanel; import docking.widgets.fieldpanel.internal.FieldPanelCoordinator; import docking.widgets.fieldpanel.support.FieldLocation; @@ -39,7 +40,7 @@ import ghidra.program.model.address.AddressSetView; import ghidra.program.model.listing.*; import ghidra.program.util.FunctionUtility; import ghidra.program.util.ProgramLocation; -import ghidra.util.HTMLUtilities; +import ghidra.util.*; /** * Panel that displays two decompilers for comparison @@ -58,7 +59,6 @@ public abstract class DecompilerCodeComparisonPanel dualDecompileResultsListenerList = @@ -70,6 +70,10 @@ public abstract class DecompilerCodeComparisonPanel rightDecompileDataSet(decompileData)); + DecompilerController leftController = cPanels[LEFT].getController(); + leftProgramListener = new DecompilerProgramListener(leftController, () -> refresh(LEFT)); + DecompilerController rightController = cPanels[RIGHT].getController(); + rightProgramListener = new DecompilerProgramListener(rightController, () -> refresh(RIGHT)); + leftDecompilerLocationListener = (leftLocation, trigger) -> { if (dualDecompilerCoordinator != null) { dualDecompilerCoordinator.leftLocationChanged(leftLocation); @@ -291,6 +300,14 @@ public abstract class DecompilerCodeComparisonPanel> getPanelThisSupersedes(); - @Override public ActionContext getActionContext(ComponentProvider provider, MouseEvent event) { @@ -594,6 +631,7 @@ public abstract class DecompilerCodeComparisonPanel toRemove = new HashSet<>(); + for (DualDecompileResultsListener l : dualDecompileResultsListenerList) { + if (MyDecompileResultsListener.class.isInstance(l)) { + toRemove.add((DecompilerCodeComparisonPanel.MyDecompileResultsListener) l); + } + } + dualDecompileResultsListenerList.removeAll(toRemove); // Clear the left or right function by passing null to the load method // and then reload it below to get it to update. @@ -726,12 +771,13 @@ public abstract class DecompilerCodeComparisonPanelToggle the layout of the decompiler " + @@ -749,4 +795,26 @@ public abstract class DecompilerCodeComparisonPanel iter = ev.iterator(); + while (iter.hasNext()) { + DomainObjectChangeRecord record = iter.next(); + if (record.getEventType() == DomainObject.DO_PROPERTY_CHANGED) { + if (record.getOldValue() instanceof String) { + String value = (String) record.getOldValue(); + if (value.startsWith(SpecExtension.SPEC_EXTENSION)) { + controller.resetDecompiler(); + break; + } + } + } + } + } + + updater.update(); + } + + public void dispose() { + updater.dispose(); + } +} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DualDecompilerActionContext.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DualDecompilerActionContext.java index 9625aaad93..5a4ae7abe5 100644 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DualDecompilerActionContext.java +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/component/DualDecompilerActionContext.java @@ -18,17 +18,17 @@ package ghidra.app.decompiler.component; import java.awt.Component; import docking.ComponentProvider; -import docking.DefaultActionContext; import docking.widgets.fieldpanel.internal.FieldPanelCoordinator; import ghidra.app.context.RestrictedAddressSetContext; +import ghidra.app.util.viewer.util.CodeComparisonActionContext; import ghidra.app.util.viewer.util.CodeComparisonPanel; -import ghidra.app.util.viewer.util.CodeComparisonPanelActionContext; +import ghidra.program.model.listing.Function; /** * Action context for a dual decompiler panel. */ -public class DualDecompilerActionContext extends DefaultActionContext - implements RestrictedAddressSetContext, CodeComparisonPanelActionContext { +public class DualDecompilerActionContext extends CodeComparisonActionContext + implements RestrictedAddressSetContext { private CodeComparisonPanel codeComparisonPanel = null; @@ -56,4 +56,20 @@ public class DualDecompilerActionContext extends DefaultActionContext public CodeComparisonPanel getCodeComparisonPanel() { return codeComparisonPanel; } + + @Override + public Function getSourceFunction() { + boolean leftHasFocus = codeComparisonPanel.leftPanelHasFocus(); + + return leftHasFocus ? codeComparisonPanel.getRightFunction() + : codeComparisonPanel.getLeftFunction(); + } + + @Override + public Function getTargetFunction() { + boolean leftHasFocus = codeComparisonPanel.leftPanelHasFocus(); + + return leftHasFocus ? codeComparisonPanel.getLeftFunction() + : codeComparisonPanel.getRightFunction(); + } } diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/BlockSignature.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/BlockSignature.java new file mode 100644 index 0000000000..30ee6d5f38 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/BlockSignature.java @@ -0,0 +1,85 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.decompiler.signature; + +import static ghidra.program.model.pcode.AttributeId.*; + +import ghidra.program.model.address.Address; +import ghidra.program.model.lang.Language; +import ghidra.program.model.pcode.*; + +/** + * A feature rooted in a basic block. There are two forms of a block feature. + * Form 1 contains only local control-flow information about the basic block. + * Form 2 is a feature that combines two operations that occur in sequence within the block. + * This form incorporates info about the operations and data-flow info about their inputs. + */ +public class BlockSignature extends DebugSignature { + public Address blockSeq; // Address of the (start of) the basic block producing this signature + public int index; // The basic block's index + public SequenceNumber opSeq; // Address of the primary operation producing this signature + public String opcode; // Op-code of the primary operation + public SequenceNumber previousOpSeq; // Address of root operation previous to primary + public String previousOpcode; // Op-code of the previous operation + + @Override + public void decode(Decoder decoder) throws DecoderException { + opSeq = null; + opcode = null; + previousOpcode = null; + previousOpSeq = null; + int pos = 0; + int el = decoder.openElement(); + hash = (int) decoder.readUnsignedInteger(ATTRIB_HASH); + index = (int) decoder.readSignedInteger(ATTRIB_INDEX); + blockSeq = AddressXML.decode(decoder); + for (;;) { + int subel = decoder.openElement(); + if (subel == 0) { + break; + } + int opc = (int) decoder.readSignedInteger(ATTRIB_CODE); + String currentOpcode = PcodeOp.getMnemonic(opc); + SequenceNumber currentOpSeq = SequenceNumber.decode(decoder); + decoder.closeElementSkipping(subel); + if (pos == 0) { + opSeq = currentOpSeq; + opcode = currentOpcode; + } + else { + previousOpSeq = currentOpSeq; + previousOpcode = currentOpcode; + } + pos += 1; + } + decoder.closeElement(el); + } + + @Override + public void printRaw(Language language, StringBuffer buf) { + buf.append(Integer.toHexString(hash)); + buf.append(" - block "); + buf.append(blockSeq.toString()); + if (previousOpcode != null) { + buf.append(" - op="); + buf.append(previousOpcode).append(" ").append(previousOpSeq.toString()); + } + if (opcode != null) { + buf.append(" - op="); + buf.append(opcode).append(" ").append(opSeq.toString()); + } + } +} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/CopySignature.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/CopySignature.java new file mode 100644 index 0000000000..2effebd9b3 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/CopySignature.java @@ -0,0 +1,50 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.decompiler.signature; + +import static ghidra.program.model.pcode.AttributeId.*; +import static ghidra.program.model.pcode.ElementId.*; + +import ghidra.program.model.lang.Language; +import ghidra.program.model.pcode.Decoder; +import ghidra.program.model.pcode.DecoderException; + +/** + * A feature representing 1 or more "stand-alone" copies in a basic block. + * A COPY operation is considered stand-alone if either a constant or a function input + * is copied into a location that is then not read directly by the function. + * These COPYs are incorporated into a single feature, which encodes the number + * and type of COPYs but does not encode the order in which they occur within the block. + */ +public class CopySignature extends DebugSignature { + public int index; // The basic block's index + + @Override + public void decode(Decoder decoder) throws DecoderException { + int el = decoder.openElement(ELEM_COPYSIG); + hash = (int) decoder.readUnsignedInteger(ATTRIB_HASH); + index = (int) decoder.readSignedInteger(ATTRIB_INDEX); + decoder.closeElement(el); + } + + @Override + public void printRaw(Language language, StringBuffer buf) { + buf.append(Integer.toHexString(hash)); + buf.append(" - Copies in block "); + buf.append(index); + } + +} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/DebugSignature.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/DebugSignature.java new file mode 100644 index 0000000000..11c853d380 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/DebugSignature.java @@ -0,0 +1,84 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.decompiler.signature; + +import static ghidra.program.model.pcode.ElementId.*; + +import java.util.ArrayList; + +import ghidra.program.model.lang.Language; +import ghidra.program.model.listing.Function; +import ghidra.program.model.pcode.Decoder; +import ghidra.program.model.pcode.DecoderException; + +/** + * A feature extracted from a function, with an additional description of what information is + * incorporated into the feature. The feature may incorporate data-flow and/or control-flow + * information from the function. Internally, the feature is a 32-bit hash of this information, but + * derived classes from this abstract class include more detailed information about how the hash was formed. + */ +public abstract class DebugSignature { + public int hash; // The underlying 32-bit hash of the feature + + /** + * Decode the feature from a stream. + * @param decoder is the stream decoder + * @throws DecoderException for problems reading the stream + */ + public abstract void decode(Decoder decoder) throws DecoderException; + + /** + * Write a brief description of this feature to the given StringBuffer. + * @param language is the underlying language of the function + * @param buf is the given StringBuffer + */ + public abstract void printRaw(Language language, StringBuffer buf); + + /** + * Decode an array of features from the stream. Collectively, the features make up + * a "feature vector" for a specific function. Each feature is returned as a separate descriptive object. + * @param decoder is the stream decoder + * @param func is the specific function whose feature vector is being decoded + * @return the array of feature objects + * @throws DecoderException for problems reading from the stream + */ + public static ArrayList decodeSignatures(Decoder decoder, Function func) + throws DecoderException { + ArrayList res = new ArrayList<>(); + int el = decoder.openElement(); + int subel = decoder.peekElement(); + while (subel != 0) { + DebugSignature sig; + if (subel == ELEM_VARSIG.id()) { + sig = new VarnodeSignature(); + } + else if (subel == ELEM_BLOCKSIG.id()) { + sig = new BlockSignature(); + } + else if (subel == ELEM_COPYSIG.id()) { + sig = new CopySignature(); + } + else { + throw new DecoderException("Unknown debug signature element"); + } + sig.decode(decoder); + res.add(sig); + subel = decoder.peekElement(); + } + decoder.closeElement(el); + return res; + } +} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/SignatureResult.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/SignatureResult.java new file mode 100644 index 0000000000..68468f8e20 --- /dev/null +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/SignatureResult.java @@ -0,0 +1,104 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.decompiler.signature; + +import static ghidra.program.model.pcode.AttributeId.*; +import static ghidra.program.model.pcode.ElementId.*; + +import java.util.ArrayList; + +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; +import ghidra.program.model.pcode.*; + +/** + * An unordered list of features describing a single function. + * Each feature represents partial information about the control-flow and/or data-flow + * making up the function. Together the features form an (approximately) complete representation + * of the function. Each feature is represented internally as 32-bit hash. Details of how the + * feature was formed are not available through this object, but see {@link DebugSignature} + * This object may optionally include a list of addresses of functions directly called by + * the function being described. + */ +public class SignatureResult { + public int[] features; // Raw features of the result + public ArrayList

calllist; // List of addresses being called + public boolean hasunimplemented; // Function has unimplemented instructions + public boolean hasbaddata; // Instruction flow went into baddata + + /** + * Decode a sequence of raw feature hashes associated with a specific function from a stream. + * The stream may optionally include addresses of called functions. + * @param decoder is the stream decoder + * @param func is the specific function being described + * @param keepcalllist is true if call addresses should be stored in the result object + * @return the decoded SignatureResult + * @throws DecoderException for problems reading from the stream + */ + public static SignatureResult decode(Decoder decoder, Function func, boolean keepcalllist) + throws DecoderException { + ArrayList res = null; + ArrayList
calllist = null; + boolean hasunimpl = false; + boolean hasbaddata = false; + if (keepcalllist) { + calllist = new ArrayList<>(); + } + int start = decoder.openElement(ELEM_SIGNATURES); + for (;;) { + int attribId = decoder.getNextAttributeId(); + if (attribId == 0) { + break; + } + if (attribId == ATTRIB_UNIMPL.id()) { + hasunimpl = decoder.readBool(); + } + else if (attribId == ATTRIB_BADDATA.id()) { + hasbaddata = decoder.readBool(); + } + } + res = new ArrayList<>(); + for (;;) { + int subel = decoder.openElement(); + if (subel == 0) { + break; + } + if (subel == ELEM_SIG.id()) { + int val = (int) decoder.readUnsignedInteger(ATTRIB_VAL); + res.add(val); + } + else { + Address addr = AddressXML.decodeFromAttributes(decoder); + if (keepcalllist) { + calllist.add(addr); + } + } + decoder.closeElement(subel); + } + decoder.closeElement(start); + + SignatureResult sigres = new SignatureResult(); + sigres.calllist = calllist; + sigres.hasunimplemented = hasunimpl; + sigres.hasbaddata = hasbaddata; + sigres.features = new int[res.size()]; + for (int i = 0; i < res.size(); ++i) { + sigres.features[i] = res.get(i).intValue(); + } + return sigres; + } + +} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/VarnodeSignature.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/VarnodeSignature.java new file mode 100644 index 0000000000..863ffeb50b --- /dev/null +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/decompiler/signature/VarnodeSignature.java @@ -0,0 +1,78 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.app.decompiler.signature; + +import static ghidra.program.model.pcode.AttributeId.*; +import static ghidra.program.model.pcode.ElementId.*; + +import ghidra.program.model.address.Address; +import ghidra.program.model.lang.Language; +import ghidra.program.model.pcode.*; + +/** + * A feature representing a portion of the data-flow graph rooted at a particular Varnode. + * The feature recursively incorporates details about the Varnode, the operation that defined it, and + * the operation's input Varnodes, up to a specific depth. + */ +public class VarnodeSignature extends DebugSignature { + public Varnode vn; // Root of the data-flow feature + public SequenceNumber seqNum; // The sequence number of the operation defining the root (may be null) + public String opcode; // The name of the defining operation (may be null) + + @Override + public void decode(Decoder decoder) throws DecoderException { + int el = decoder.openElement(ELEM_VARSIG); + hash = (int) decoder.readUnsignedInteger(ATTRIB_HASH); + int subel = decoder.openElement(); + Address vnAddr = AddressXML.decodeFromAttributes(decoder); + int vnSize = 0; + if (vnAddr.getAddressSpace().isVariableSpace()) { + //varnodes in the variable space will have a default offset of -1 + //but we can get the correct size + decoder.rewindAttributes(); + Varnode.Join join = Varnode.decodePieces(decoder); + vnSize = join.logicalSize; + } + else { + vnSize = (int) decoder.readSignedInteger(ATTRIB_SIZE); + } + vn = new Varnode(vnAddr, vnSize); + decoder.closeElement(subel); + if (decoder.peekElement() != 0) { + subel = decoder.openElement(); + int opc = (int) decoder.readSignedInteger(ATTRIB_CODE); + opcode = PcodeOp.getMnemonic(opc); + seqNum = SequenceNumber.decode(decoder); + decoder.closeElementSkipping(subel); // Skip the input/output varnodes + } + else { + seqNum = null; + opcode = null; + } + decoder.closeElement(el); + } + + @Override + public void printRaw(Language language, StringBuffer buf) { + buf.append(Integer.toHexString(hash)); + buf.append(" - var "); + buf.append(vn.toString(language)); + if (seqNum != null) { + buf.append(" - op="); + buf.append(opcode).append(" ").append(seqNum.toString()); + } + } +} diff --git a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/plugin/core/decompile/DecompilerProvider.java b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/plugin/core/decompile/DecompilerProvider.java index 8e990744a4..40a88228be 100644 --- a/Ghidra/Features/Decompiler/src/main/java/ghidra/app/plugin/core/decompile/DecompilerProvider.java +++ b/Ghidra/Features/Decompiler/src/main/java/ghidra/app/plugin/core/decompile/DecompilerProvider.java @@ -38,16 +38,15 @@ import ghidra.app.plugin.core.decompile.actions.*; import ghidra.app.services.*; import ghidra.app.util.HelpTopics; import ghidra.app.util.ListingHighlightProvider; -import ghidra.framework.model.*; import ghidra.framework.options.*; import ghidra.framework.plugintool.NavigatableComponentProviderAdapter; import ghidra.framework.plugintool.util.ServiceListener; -import ghidra.program.database.SpecExtension; import ghidra.program.model.address.*; import ghidra.program.model.listing.Function; import ghidra.program.model.listing.Program; import ghidra.program.model.symbol.*; -import ghidra.program.util.*; +import ghidra.program.util.ProgramLocation; +import ghidra.program.util.ProgramSelection; import ghidra.util.HelpLocation; import ghidra.util.Swing; import ghidra.util.bean.field.AnnotatedTextFieldElement; @@ -57,8 +56,8 @@ import resources.MultiIconBuilder; import utility.function.Callback; public class DecompilerProvider extends NavigatableComponentProviderAdapter - implements DomainObjectListener, OptionsChangeListener, DecompilerCallbackHandler, - DecompilerHighlightService, DecompilerMarginService { + implements OptionsChangeListener, DecompilerCallbackHandler, DecompilerHighlightService, + DecompilerMarginService { private static final String OPTIONS_TITLE = "Decompiler"; @@ -101,6 +100,7 @@ public class DecompilerProvider extends NavigatableComponentProviderAdapter private ViewerPosition pendingViewerPosition; private SwingUpdateManager redecompileUpdater; + private DecompilerProgramListener programListener; // Follow-up work can be items that need to happen after a pending decompile is finished, such // as updating highlights after a variable rename @@ -164,6 +164,7 @@ public class DecompilerProvider extends NavigatableComponentProviderAdapter followUpWorkUpdater = new SwingUpdateManager(() -> doFollowUpWork()); plugin.getTool().addServiceListener(serviceListener); + programListener = new DecompilerProgramListener(controller, redecompileUpdater); } //================================================================================================== @@ -310,38 +311,10 @@ public class DecompilerProvider extends NavigatableComponentProviderAdapter // DomainObjectListener methods //================================================================================================== - @Override - public void domainObjectChanged(DomainObjectChangedEvent ev) { - // Check for events that signal that a decompiler process' data is stale - // and if so force a new process to be spawned - if (ev.containsEvent(ChangeManager.DOCR_MEMORY_BLOCK_ADDED) || - ev.containsEvent(ChangeManager.DOCR_MEMORY_BLOCK_REMOVED) || - ev.containsEvent(DomainObject.DO_OBJECT_RESTORED)) { - controller.resetDecompiler(); - } - else if (ev.containsEvent(DomainObject.DO_PROPERTY_CHANGED)) { - Iterator iter = ev.iterator(); - while (iter.hasNext()) { - DomainObjectChangeRecord record = iter.next(); - if (record.getEventType() == DomainObject.DO_PROPERTY_CHANGED) { - if (record.getOldValue() instanceof String) { - String value = (String) record.getOldValue(); - if (value.startsWith(SpecExtension.SPEC_EXTENSION)) { - controller.resetDecompiler(); - break; - } - } - } - } - } - - // Trigger a redecompile an any program change if the window is active - if (isVisible()) { - redecompileUpdater.update(); - } - } - private void doRefresh(boolean optionsChanged) { + if (!isVisible()) { + return; + } ToolOptions fieldOptions = tool.getOptions(GhidraOptions.CATEGORY_BROWSER_FIELDS); ToolOptions opt = tool.getOptions(OPTIONS_TITLE); @@ -441,14 +414,14 @@ public class DecompilerProvider extends NavigatableComponentProviderAdapter void doSetProgram(Program newProgram) { controller.clear(); if (program != null) { - program.removeListener(this); + program.removeListener(programListener); } program = newProgram; currentLocation = null; currentSelection = null; if (program != null) { - program.addListener(this); + program.addListener(programListener); ToolOptions fieldOptions = tool.getOptions(GhidraOptions.CATEGORY_BROWSER_FIELDS); ToolOptions opt = tool.getOptions(OPTIONS_TITLE); decompilerOptions.grabFromToolAndProgram(fieldOptions, opt, program); diff --git a/Ghidra/Features/VersionTracking/src/main/java/ghidra/feature/vt/api/correlator/program/VTAbstractReferenceProgramCorrelator.java b/Ghidra/Features/VersionTracking/src/main/java/ghidra/feature/vt/api/correlator/program/VTAbstractReferenceProgramCorrelator.java index 30cc6f09cd..e659c603d4 100644 --- a/Ghidra/Features/VersionTracking/src/main/java/ghidra/feature/vt/api/correlator/program/VTAbstractReferenceProgramCorrelator.java +++ b/Ghidra/Features/VersionTracking/src/main/java/ghidra/feature/vt/api/correlator/program/VTAbstractReferenceProgramCorrelator.java @@ -393,7 +393,7 @@ public abstract class VTAbstractReferenceProgramCorrelator extends VTAbstractPro Counter totalMatches = new Counter(); Collection matchSets = getMatchSets(matchSet.getSession(), totalMatches); - monitor.initialize(totalMatches.count); + monitor.initialize(totalMatches.count()); // Loop through the matchSets in order to get total source and destination reference // counts that pass the filter @@ -471,7 +471,7 @@ public abstract class VTAbstractReferenceProgramCorrelator extends VTAbstractPro } dedupedMatchSets.put(name, ms); - totalMatches.count += ms.getMatchCount(); + totalMatches.add(ms.getMatchCount()); } return dedupedMatchSets.values(); diff --git a/Ghidra/Features/VersionTrackingBSim/Module.manifest b/Ghidra/Features/VersionTrackingBSim/Module.manifest new file mode 100755 index 0000000000..e69de29bb2 diff --git a/Ghidra/Features/VersionTrackingBSim/build.gradle b/Ghidra/Features/VersionTrackingBSim/build.gradle new file mode 100755 index 0000000000..351da68da4 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/build.gradle @@ -0,0 +1,31 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +apply from: "$rootProject.projectDir/gradle/distributableGhidraModule.gradle" +apply from: "$rootProject.projectDir/gradle/javaProject.gradle" +apply from: "$rootProject.projectDir/gradle/javaTestProject.gradle" +apply from: "$rootProject.projectDir/gradle/helpProject.gradle" +apply plugin: 'eclipse' +eclipse.project.name = 'Features VersionTrackingBSim' + +dependencies { + api project(":Decompiler") + api project(":BSim") + api project(":VersionTracking") + + testImplementation project(path: ':VersionTracking', configuration: 'testArtifacts') + testImplementation project(path: ':VersionTracking', configuration: 'testArtifacts') + integrationTestImplementation project(':VersionTracking') +} diff --git a/Ghidra/Features/VersionTrackingBSim/certification.manifest b/Ghidra/Features/VersionTrackingBSim/certification.manifest new file mode 100755 index 0000000000..5b5390cfde --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/certification.manifest @@ -0,0 +1,5 @@ +##VERSION: 2.0 +##MODULE IP: LGPL 3.0 +Module.manifest||GHIDRA||||END| +src/main/help/help/TOC_Source.xml||GHIDRA||||END| +src/main/help/help/topics/BSimCorrelator/BSim_Correlator.html||GHIDRA||||END| diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/help/help/TOC_Source.xml b/Ghidra/Features/VersionTrackingBSim/src/main/help/help/TOC_Source.xml new file mode 100755 index 0000000000..9a0ef2c520 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/help/help/TOC_Source.xml @@ -0,0 +1,67 @@ + + + + + + + + + + + + + + + + + + + diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/help/help/topics/BSimCorrelator/BSim_Correlator.html b/Ghidra/Features/VersionTrackingBSim/src/main/help/help/topics/BSimCorrelator/BSim_Correlator.html new file mode 100755 index 0000000000..fa711880cb --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/help/help/topics/BSimCorrelator/BSim_Correlator.html @@ -0,0 +1,99 @@ + + + + + + + BSim Program Correlator + + + + + +

BSim Program Correlator

+ +
+

The BSim Program + Correlator uses the decompiler to generate confidence scores between potentially matching + functions in the source and destination programs. Function call-graphs are used to further + boost the scores and distinguish between conflicting matches. + .

+ +

The decompiler generates a formal feature vector for a function, where individual features + are extracted from the control-flow and data-flow characteristics of its normalized p-code + representation.

+ +

Functions are compared by comparing their corresponding feature vectors, from which + similarity and confidence scores are extracted.

+ +

A confidence score, for this correlator, is an open-ended floating-point value + (ranging from -infinity to +infinity) describing the amount of correspondence between the + control-flow and data-flow of two functions. A good working range for setting thresholds + (below) and for describing function pairs with some matching features is 0.0 to 100.0. + A score of 0.0 corresponds to functions with roughly equal amounts of similar and dissimilar features. + A score of 10.0 is typical for small identical functions, and 100.0 is achieved by pairs + of larger sized identical functions.

+ +

The correlator initially collects high confidence (high scoring) matches as a "seed" set. + Then, using call-graph information, the seed matches are extended to additional matches + throughout the programs.

+ +

There are four options for the BSim Program Correlator:

+ +

Confidence Threshold for a Match

+ +
+

This option sets the threshold for accepting + a new match by following the call-graph from a previously accepted pair of matching functions. + Because potential pairs are drawn from the local call-graph neighborhood of an + accepted pair, this threshold is typically set lower than the seed threshold.

+
+ +

Confidence Threshold for a Seed

+ +
+

This establishes the threshold for choosing + potential matches as part of the initial "seed" set. Be careful setting this threshold + lower than the default, as any false match in the initial seed set is more likely to propagate.

+
+ +

Memory Model

+ +
+

The memory model option selects how much memory to use for finding + matches. If you run out of memory correlating large programs, lower this choice to "Medium" + or "Small"...note however that correlation may be slightly less accurate.

+
+ +

Use Accepted Matches as Seeds

+ +
+

This option indicates whether to include + previously accepted matches, typically from other correlators, into the initial "seed" set. + The BSim Program Correlator will still try to find additional seed matches to merge + with the already accepted matches. If you want to only use the incoming accepted + matches, set the Confidence Threshold for a Seed extremely high (like 99999999 or + so). Be careful to accept only high confidence matches prior to using this option, as + any errors in the initial seed set are more likely to propagate.

+
+ +
+ +

Related Topics:

+ +
+
+ + diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelator.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelator.java new file mode 100755 index 0000000000..dcfa7b3fd5 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelator.java @@ -0,0 +1,350 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import java.io.IOException; +import java.io.InputStream; +import java.util.ArrayList; +import java.util.List; + +import generic.cache.CachingPool; +import generic.cache.CountingBasicFactory; +import generic.concurrent.QCallback; +import generic.jar.ResourceFile; +import generic.lsh.LSHMemoryModel; +import generic.lsh.vector.*; +import ghidra.app.decompiler.*; +import ghidra.app.decompiler.parallel.ParallelDecompiler; +import ghidra.app.decompiler.signature.SignatureResult; +import ghidra.feature.vt.api.main.VTMatchInfo; +import ghidra.feature.vt.api.main.VTMatchSet; +import ghidra.feature.vt.api.util.VTAbstractProgramCorrelator; +import ghidra.feature.vt.api.util.VTFunctionSizeUtil; +import ghidra.features.bsim.query.GenSignatures; +import ghidra.framework.options.ToolOptions; +import ghidra.program.model.address.*; +import ghidra.program.model.lang.CompilerSpec.EvaluationModelType; +import ghidra.program.model.lang.LanguageID; +import ghidra.program.model.lang.PrototypeModel; +import ghidra.program.model.listing.*; +import ghidra.program.model.symbol.Reference; +import ghidra.program.model.symbol.ReferenceManager; +import ghidra.util.Msg; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; +import ghidra.util.xml.SpecXmlUtils; +import ghidra.xml.NonThreadedXmlPullParserImpl; +import ghidra.xml.XmlPullParser; + +/** + * Correlator which discovers functional matches by comparing data-flow feature vectors. + * An initial seed set of high confidence matches are chosen. The match set is extended + * from the seeds by using local neighborhoods around the accepted match to efficiently + * discover new matches. + */ +public class BSimProgramCorrelator extends VTAbstractProgramCorrelator { + + private LSHVectorFactory vectorFactory; + private static final int TIMEOUT = 60; + public static final double SIMILARITY_THRESHOLD = 0.5; + // note that the utils function strips out thunks now so we just set + // minimum size to 0 assuming call graph will save us + public static final int FUNCTION_MINIMUM_SIZE = 0; + + protected BSimProgramCorrelator(Program sourceProgram, AddressSetView sourceAddressSet, + Program destinationProgram, AddressSetView destinationAddressSet, ToolOptions options) { + super(sourceProgram, sourceAddressSet, destinationProgram, destinationAddressSet, options); + vectorFactory = new WeightedLSHCosineVectorFactory(); + } + + @Override + public String getName() { + return BSimProgramCorrelatorFactory.NAME; + } + + @Override + protected void doCorrelate(VTMatchSet matchSet, TaskMonitor monitor) throws CancelledException { + ToolOptions options = getOptions(); + LSHMemoryModel model = options.getEnum(BSimProgramCorrelatorFactory.MEMORY_MODEL, + BSimProgramCorrelatorFactory.MEMORY_MODEL_DEFAULT); + + double confThreshold = options.getDouble(BSimProgramCorrelatorFactory.SEED_CONF_THRESHOLD, + BSimProgramCorrelatorFactory.SEED_CONF_THRESHOLD_DEFAULT); + double impThreshold = options.getDouble(BSimProgramCorrelatorFactory.IMPLICATION_THRESHOLD, + BSimProgramCorrelatorFactory.IMPLICATION_THRESHOLD_DEFAULT); + + boolean useAcceptedMatchesAsSeeds = + options.getBoolean(BSimProgramCorrelatorFactory.USE_ACCEPTED_MATCHES_AS_SEEDS, + BSimProgramCorrelatorFactory.USE_ACCEPTED_MATCHES_AS_SEEDS_DEFAULT); + + boolean useNamespace = false; // By default we don't have namespace info + boolean useCallRefs = false; // By default we use decompiler to generate callgraph + + List result; + try { + LanguageID id1 = getSourceProgram().getLanguageID(); + LanguageID id2 = getDestinationProgram().getLanguageID(); + //Use special weights for LSHCosineVectors + ResourceFile defaultWeightsFile = GenSignatures.getWeightsFile(id1, id2); + if (defaultWeightsFile == null) { + + // known limitation; hoped to be fixed in the future + Msg.showWarn(this, null, "Cannot Compare Programs", + "Cannot currently compare programs with such different architectures.
" + + "Source program is " + id1.getIdAsString() + "
" + + "Destination program is " + id2.getIdAsString()); + return; + } + if (defaultWeightsFile.getName().contains("cpool")) { + // With constant pool languages (Dalvik, JVM) + useNamespace = true; // We have reliable namespace info + useCallRefs = true; // We don't have absolute calls, use references + } + InputStream input = defaultWeightsFile.getInputStream(); + XmlPullParser parser = new NonThreadedXmlPullParserImpl(input, "Vector weights parser", + SpecXmlUtils.getXmlHandler(), false); + vectorFactory.readWeights(parser); + input.close(); + + monitor.setMessage("Generating source dictionary"); + List rawSourceNodes = + generateNodes(getSourceProgram(), getSourceAddressSet(), useCallRefs, monitor); + FunctionNodeContainer sourceNodes = + new FunctionNodeContainer(getSourceProgram(), rawSourceNodes); + + monitor.setMessage("Generating destination dictionary"); + List rawDestNodes = generateNodes(getDestinationProgram(), + getDestinationAddressSet(), useCallRefs, monitor); + FunctionNodeContainer destNodes = + new FunctionNodeContainer(getDestinationProgram(), rawDestNodes); + + BSimProgramCorrelatorMatching omni = + new BSimProgramCorrelatorMatching(sourceNodes, destNodes, vectorFactory, + confThreshold, impThreshold, SIMILARITY_THRESHOLD, useNamespace, model); + omni.discoverPotentialMatches(monitor); + if (!omni.generateSeeds(matchSet, useAcceptedMatchesAsSeeds, monitor)) { + Msg.info(this, "BSim Program Correlator could not find any seeds"); + } + result = omni.doMatching(monitor); //Do the matching! + } + catch (InterruptedException e) { + Msg.error(this, "Error Correlating", e.getCause()); + CancelledException cancelledException = new CancelledException(); + cancelledException.initCause(e); + throw cancelledException; + } + catch (CancelledException ce) { + throw ce; + } + catch (Exception e) { + Msg.error(this, "Error Correlating", e.getCause()); + CancelledException cancelledException = new CancelledException(); + cancelledException.initCause(e); + throw cancelledException; + } + + wrapUp(result, matchSet, monitor); // Display matches, print stuff, etc. + + return; + } + + private static void addExternalFunctions(Program program, List list, + LSHVectorFactory vFactory, TaskMonitor monitor) throws CancelledException { + FunctionIterator iter = program.getFunctionManager().getExternalFunctions(); + // Create a generic feature vector to represent external functions + int[] externalFeatures = new int[1]; + externalFeatures[0] = 0xfade5eed; + LSHVector externalVector = vFactory.buildVector(externalFeatures); + while (iter.hasNext()) { + monitor.checkCancelled(); + Function func = iter.next(); + FunctionNode node = new FunctionNode(func, externalVector, new ArrayList
()); + list.add(node); + } + } + + private List generateNodes(final Program program, AddressSetView addrSet, + boolean useCallRefs, final TaskMonitor monitor) + throws InterruptedException, CancelledException, Exception { + + monitor.checkCancelled(); + + CachingPool decompilerPool = new CachingPool( + new DecompilerFactory(program, vectorFactory.getSettings())); + ParallelDecompilerCallback callback = + new ParallelDecompilerCallback(decompilerPool, vectorFactory, useCallRefs); + + List results = null; + try { + AddressSetView refinedAddressSet = VTFunctionSizeUtil.minimumSizeFunctionFilter(program, + addrSet, FUNCTION_MINIMUM_SIZE, monitor); + results = ParallelDecompiler.decompileFunctions(callback, program, refinedAddressSet, + monitor); + } + finally { + decompilerPool.dispose(); + } + + addExternalFunctions(program, results, vectorFactory, monitor); + + monitor.setMessage("Collecting dictionary results"); + return results; + } + + private static void wrapUp(List result, final VTMatchSet matchSet, + final TaskMonitor monitor) throws CancelledException { + + //Populate the table with matches. + monitor.setMessage("Adding results to database"); + monitor.setIndeterminate(false); + monitor.initialize(result.size()); + int ii = 0; + for (FunctionPair resMatch : result) { + VTMatchInfo match = resMatch.getMatch(matchSet); + ++ii; + if (ii % 1000 == 0) { + monitor.checkCancelled(); + monitor.incrementProgress(1000); + } + matchSet.addMatch(match); + } + return; + } + + /** + * Establish decompiler options for the feature vector calculation + * @param program is the specific program to decompile + * @return the formal options object + */ + private static DecompileOptions getDecompilerOptions(Program program) { + DecompileOptions options = new DecompileOptions(); + options.setNoCastPrint(true); + try { + final PrototypeModel model = program.getCompilerSpec() + .getPrototypeEvaluationModel(EvaluationModelType.EVAL_CURRENT); + options.setProtoEvalModel(model.getName()); + } + catch (Exception e) { + Msg.warn(BSimProgramCorrelator.class, + "problem setting prototype evaluation model: " + e.getMessage()); + } + options.setDefaultTimeout(TIMEOUT); + return options; + } + +//================================================================================================== +// Inner Classes +//================================================================================================== + + private static class DecompilerFactory extends CountingBasicFactory { + + private Program program; + private int settings; + + DecompilerFactory(Program program, int set) { + this.program = program; + settings = set; + } + + @Override + public DecompInterface doCreate(int itemNumber) throws IOException { + DecompInterface decompiler = new DecompInterface(); + decompiler.setOptions(getDecompilerOptions(program)); + decompiler.setSignatureSettings(settings); + if (!decompiler.openProgram(program)) { + throw new IOException(decompiler.getLastMessage()); + } + return decompiler; + } + + @Override + public void doDispose(DecompInterface decompiler) { + decompiler.dispose(); + } + } + + private static class ParallelDecompilerCallback implements QCallback { + + private LSHVectorFactory vectorFactory; + private CachingPool pool; + private boolean callsByReference; + + ParallelDecompilerCallback(CachingPool decompilerPool, + LSHVectorFactory vFactory, boolean refCalls) { + vectorFactory = vFactory; + this.pool = decompilerPool; + callsByReference = refCalls; + } + + private ArrayList
getCallAddressesByReference(Function function, + TaskMonitor monitor) throws CancelledException { + ArrayList
resultList = new ArrayList
(); + Program program = function.getProgram(); + ReferenceManager referenceManager = program.getReferenceManager(); + AddressSetView addresses = function.getBody(); + AddressIterator addressIterator = addresses.getAddresses(true); + while (addressIterator.hasNext()) { + monitor.checkCancelled(); + Address address = addressIterator.next(); + Reference[] referencesFrom = referenceManager.getReferencesFrom(address); + if (referencesFrom != null) { + for (Reference reference : referencesFrom) { + if (reference.getReferenceType().isCall()) { + resultList.add(reference.getToAddress()); + } + } + } + } + return resultList; + } + + @Override + public FunctionNode process(Function function, TaskMonitor monitor) throws Exception { + + monitor.checkCancelled(); + DecompInterface decompiler = pool.get(); + try { + LSHVector vec = null; + ArrayList
callAddresses = null; + SignatureResult sigres = + decompiler.generateSignatures(function, !callsByReference, TIMEOUT, monitor); + if (sigres == null) { + callAddresses = new ArrayList
(); + } + else { + vec = vectorFactory.buildVector(sigres.features); + if (callsByReference) { + callAddresses = getCallAddressesByReference(function, monitor); + } + else { + callAddresses = sigres.calllist; //It will take a second pass through the data to figure out how the call graph fits together. + } + } + FunctionNode res = new FunctionNode(function, vec, callAddresses); + if (res.getVector() == null) { + String errmsg = decompiler.getLastMessage(); + if (errmsg.startsWith("Bad command")) { + throw new DecompileException(BSimProgramCorrelatorFactory.NAME, errmsg); + } + } + return res; + } + finally { + pool.release(decompiler); + } + } + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelatorFactory.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelatorFactory.java new file mode 100755 index 0000000000..4edec28394 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelatorFactory.java @@ -0,0 +1,105 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import generic.lsh.LSHMemoryModel; +import ghidra.feature.vt.api.main.VTProgramCorrelator; +import ghidra.feature.vt.api.main.VTProgramCorrelatorAddressRestrictionPreference; +import ghidra.feature.vt.api.util.VTAbstractProgramCorrelatorFactory; +import ghidra.feature.vt.api.util.VTOptions; +import ghidra.program.model.address.AddressSetView; +import ghidra.program.model.listing.Program; +import ghidra.util.HelpLocation; + +public class BSimProgramCorrelatorFactory extends VTAbstractProgramCorrelatorFactory { + public static final String NAME = "BSim Function Matching"; + public static final String DESC = + "Finds function matches by using data flow and call graph similarities between the " + + "source and destination programs."; + + public static final String MEMORY_MODEL = "Memory Model"; + public static final LSHMemoryModel MEMORY_MODEL_DEFAULT = LSHMemoryModel.LARGE; + public static final String MEMORY_MODEL_DESC = + "Amount of memory used to compute matches. Smaller models are slightly less accurate."; + + public static final String SEED_CONF_THRESHOLD = "Confidence Threshold for a Seed"; + public static final double SEED_CONF_THRESHOLD_DEFAULT = 10.0; + public static final String SEED_CONF_THRESHOLD_DESC = + "For threshold N, the probability that a seed is incorrect is approximately 1/2^(N/5+9)."; + + public static final String IMPLICATION_THRESHOLD = "Confidence Threshold for a Match"; + public static final double IMPLICATION_THRESHOLD_DEFAULT = 0.0; + public static final String IMPLICATION_THRESHOLD_DESC = + "For threshold N, the probability that a match is incorrect is approximately 1/2^(N/5+9)."; + + public static final String USE_ACCEPTED_MATCHES_AS_SEEDS = "Use Accepted Matches as Seeds"; + public static final boolean USE_ACCEPTED_MATCHES_AS_SEEDS_DEFAULT = true; + public static final String USE_ACCEPTED_MATCHES_AS_SEEDS_DESC = + "Already accepted matches will also be used as seeds."; + + @Override + public int getPriority() { + return 50; + } + + @Override + protected VTProgramCorrelator doCreateCorrelator(Program sourceProgram, + AddressSetView sourceAddressSet, Program destinationProgram, + AddressSetView destinationAddressSet, VTOptions options) { + return new BSimProgramCorrelator(sourceProgram, sourceAddressSet, destinationProgram, + destinationAddressSet, options); + } + + @Override + public VTProgramCorrelatorAddressRestrictionPreference getAddressRestrictionPreference() { + return VTProgramCorrelatorAddressRestrictionPreference.RESTRICTION_NOT_ALLOWED; + } + + @Override + public VTOptions createDefaultOptions() { + VTOptions options = new VTOptions(NAME); + HelpLocation help = new HelpLocation("BSimCorrelator", "BSim_Correlator"); + + options.setEnum(MEMORY_MODEL, MEMORY_MODEL_DEFAULT); + options.registerOption(MEMORY_MODEL, MEMORY_MODEL_DEFAULT, help, MEMORY_MODEL_DESC); + + options.setDouble(SEED_CONF_THRESHOLD, SEED_CONF_THRESHOLD_DEFAULT); + options.registerOption(SEED_CONF_THRESHOLD, SEED_CONF_THRESHOLD_DEFAULT, help, + SEED_CONF_THRESHOLD_DESC); + + options.setDouble(IMPLICATION_THRESHOLD, IMPLICATION_THRESHOLD_DEFAULT); + options.registerOption(IMPLICATION_THRESHOLD, IMPLICATION_THRESHOLD_DEFAULT, help, + IMPLICATION_THRESHOLD_DESC); + + options.setBoolean(USE_ACCEPTED_MATCHES_AS_SEEDS, USE_ACCEPTED_MATCHES_AS_SEEDS_DEFAULT); + options.registerOption(USE_ACCEPTED_MATCHES_AS_SEEDS, USE_ACCEPTED_MATCHES_AS_SEEDS_DEFAULT, + help, USE_ACCEPTED_MATCHES_AS_SEEDS_DESC); + + options.setOptionsHelpLocation(help); + + return options; + } + + @Override + public String getDescription() { + return DESC; + } + + @Override + public String getName() { + return NAME; + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelatorMatching.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelatorMatching.java new file mode 100755 index 0000000000..3b8211bfc9 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BSimProgramCorrelatorMatching.java @@ -0,0 +1,787 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import java.util.*; +import java.util.Map.Entry; + +import org.apache.commons.collections4.MultiValuedMap; +import org.apache.commons.collections4.multimap.HashSetValuedHashMap; + +import generic.concurrent.*; +import generic.lsh.LSHMemoryModel; +import generic.lsh.vector.LSHVectorFactory; +import generic.lsh.vector.VectorCompare; +import ghidra.feature.vt.api.NeighborGenerator.NeighborhoodPair; +import ghidra.feature.vt.api.main.*; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; +import ghidra.program.model.listing.Program; +import ghidra.util.Msg; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +/** + * Class for running the BSim function matching algorithm, which happens in stages: + * 1) Construct BSimProgramCorrelatorMatching with prepopulated FunctionNodeContainers, one for source and destination programs + * 2) Call discoverPotentialMatches to do raw vector comparisons among source and destination + * 3) Call generateSeeds to select an initial set of high confidence matches + * 4) Call doMatching to extend the seed set into a full list of matches + */ +public class BSimProgramCorrelatorMatching { + + private SortedSet implications; // Current potential matches sorted by score + private FunctionNodeContainer sourceNodes; // Nodes (functions) associated with the source program + private FunctionNodeContainer destNodes; // Nodes associated with the destination program + private LSHVectorFactory vectorFactory; // Factory for generating weighted vectors for comparing nodes + private LinkedList matches; // The list of final matches + private Set seeds; // Initial set of match pairs used for growing out full set of matches + private List discoveredMatches; // Raw of set of pairs of similar functions + private double confThreshold; // Initial confidence threshold for selecting seed matches + private double impThreshold; // Confidence threshold for extending to additional matches + private double potentialSimThreshold; // Similarity threshold used when discovering potential matches + private LSHMemoryModel memoryModel; // The memory model to use when binning vectors + private boolean useNamespaceNeighbors; // True if namespace information is used in matching + + /** + * This class is used to lookup potential matches in the {@link BinningSystem} and do + * secondary testing by computing similarities of feature vectors. + * Searching happens in parallel. + */ + private class MatchingCallback implements QCallback> { + + private BinningSystem sourceBinning; + private double simThreshold; + + MatchingCallback(BinningSystem sourceBinning, double simThreshold) { + this.sourceBinning = sourceBinning; + this.simThreshold = simThreshold; + } + + @Override + public List process(FunctionNode queryNode, TaskMonitor monitor) + throws Exception { + monitor.checkCancelled(); + + if ((queryNode == null) || (queryNode.getVector() == null)) { + monitor.incrementProgress(1); + return null; + } + + List associates = new LinkedList(); + findSimilarNodes(associates, queryNode, monitor); + monitor.incrementProgress(1); + return associates; + } + + /** + * Lookup potential matches for -queryNode- in the binning system, + * and perform secondary testing to see if we have a full (potential) match. + * Pairs that exceed the threshold are added to the -results- list + * @param results is the list of FunctionPairs passing the similarity test + * @param queryNode is the base FunctionNode to compare + * @param monitor is the TaskMonitor + * @throws CancelledException if the user cancels the correlation + */ + private void findSimilarNodes(List results, FunctionNode queryNode, + TaskMonitor monitor) throws CancelledException { + + //Set up for matching via feature vector comparison. + Set neighbors = sourceBinning.lookup(queryNode); + VectorCompare veccompare = new VectorCompare(); + + //Check each neighbor from the system of binnings to see if they pass a round of matching. + for (FunctionNode neighbor : neighbors) { + monitor.checkCancelled(); + + //Feature vector computations + double similarity = neighbor.getVector().compare(queryNode.getVector(), veccompare); + if (similarity < simThreshold) { + continue; + } + double confidence = vectorFactory.calculateSignificance(veccompare); + + //Create FunctionPair (bridge in the graph from source to dest) + FunctionPair newPair = + new FunctionPair(neighbor, queryNode, similarity, confidence); + + results.add(newPair); + } + } + } + + /** + * @param sourceNodes is the container for source functions + * @param destNodes is the container for destination functions + * @param vFactory is the factory for building feature vectors during analysis + * @param conf is the initial confidence threshold for seeds + * @param imp is the follow-on confidence for extending to additional matches + * @param sim is the similarity threshold used when discovering matches + * @param useNamespace true if namespace info is used to find additional matches + * @param model is the memory model to use when discovering seed matches + */ + public BSimProgramCorrelatorMatching(FunctionNodeContainer sourceNodes, + FunctionNodeContainer destNodes, LSHVectorFactory vFactory, double conf, double imp, + double sim, boolean useNamespace, LSHMemoryModel model) { + this.sourceNodes = sourceNodes; + this.destNodes = destNodes; + this.vectorFactory = vFactory; + confThreshold = conf; + impThreshold = imp; + potentialSimThreshold = sim; + useNamespaceNeighbors = useNamespace; + memoryModel = model; + implications = new TreeSet(); + } + + /** + * Formally accept a FunctionPair as a match. Update bookkeeping to indicate the match. + * @param bridge is the pair to accept as a match + */ + private void acceptMatch(FunctionPair bridge) { + FunctionNode sourceNode = bridge.getSourceNode(); + FunctionNode destNode = bridge.getDestNode(); + sourceNode.setAcceptedMatch(true); + destNode.setAcceptedMatch(true); + matches.add(bridge); + + // Given the pair, remove the source and destination as a potential matches from any other node. + Iterator> iter = sourceNode.getAssociateIterator(); + while (iter.hasNext()) { + iter.next().getKey().removeAssociate(sourceNode); + } + iter = destNode.getAssociateIterator(); + while (iter.hasNext()) { + iter.next().getKey().removeAssociate(destNode); + } + sourceNode.clearAssociates(); // Clear old potential matches + destNode.clearAssociates(); + } + + /** + * Do vector comparisons between the source and destination FunctionNodes. + * Anything discovered that exceeds {@link #potentialSimThreshold} is placed into {@link #discoveredMatches} + * A {@link BinningSystem} is built, then individual FunctionNodes are searched in parallel. + * @param monitor is the TaskMonitor + * @throws Exception for user cancellation or other problems + */ + public void discoverPotentialMatches(TaskMonitor monitor) throws Exception { + + BinningSystem binning = new BinningSystem(memoryModel); + monitor.setMessage("Binning source functions..."); + monitor.initialize(sourceNodes.size()); + binning.add(sourceNodes.iterator(), monitor); + + monitor.setMessage("Zealously over-pairing matches..."); + monitor.initialize(destNodes.size()); + + // + // Queue setup + // + GThreadPool pool = GThreadPool.getPrivateThreadPool("BSimProgramCorrelatorMatching"); + QCallback> callback = + new MatchingCallback(binning, potentialSimThreshold); + + // @formatter:off + ConcurrentQ> queue = + new ConcurrentQBuilder>() + .setThreadPool(pool) + .setCollectResults(true) + .setMonitor(monitor) + .build(callback); + // @formatter:on + + // + // Submit and wait for results + // + queue.add(destNodes.iterator()); + + Collection>> results; + try { + results = queue.waitForResults(); + } + finally { + queue.dispose(); + } + + discoveredMatches = new LinkedList(); + for (QResult> result : results) { + monitor.checkCancelled(); + List pieces = result.getResult(); + if (pieces == null) { + continue; + } + for (FunctionPair bridge : pieces) { + monitor.checkCancelled(); + if (bridge != null) { + FunctionNode sourceNode = bridge.getSourceNode(); + FunctionNode destNode = bridge.getDestNode(); + sourceNode.addAssociate(destNode, bridge); + destNode.addAssociate(sourceNode, bridge); + discoveredMatches.add(bridge); + } + } + } + } + + /** + * Find the last index in the (sorted) list where the confidence is >= threshold + * @param pairs is the sorted list + * @param threshold to find + * @return the index + */ + private static int findIndexMatchingThreshold(ArrayList pairs, double threshold) { + int min = 0; + int max = pairs.size() - 1; + while (min < max) { + int mid = (min + max + 1) / 2; // Guarantee if min != max, then mid != min + FunctionPair pair = pairs.get(mid); + if (pair.getConfResult() < threshold) { + max = mid - 1; + } + else { + min = mid; + } + } + return min; + } + + /** + * Choose seed FunctionNode pairs with the highest confidence from among {@link #discoveredMatches} + * making sure there are no conflicts, (a FunctionNode that is involved in multiple matches). + * Selection happens in rounds. During a round: + * a) "Accept" all pairs for which there is no immediate conflict + * b) If a pair has conflicts, throw it out if either: + * 1) The number of children is different between source and dest (difference > threshold) + * 2) The function length is different between source and dest (difference > threshold) + * + * Between rounds the "accepted" pairs and the "thrown out" pairs may remove conflicts from the + * remaining pairs. Each round the threshold for throwing out a conflict is tightened. + * + * The process terminates when no new pairs are accepted during a round. + * The accepted pairs are sorted by confidence, and those exceeding {@link #confThreshold} become + * the final seed set. + * @param monitor is the TaskMonitor + * @throws CancelledException if the user cancels the correlation + */ + private void chooseSeeds(TaskMonitor monitor) throws CancelledException { + monitor.setMessage("Generating seeds..."); + ArrayList finalPairs = new ArrayList(); + HashSet matchedSource = new HashSet(); // Source functions that are matched + HashSet matchedDest = new HashSet(); // Dest functions that are matched + MultiValuedMap sourceHoldOn = + new HashSetValuedHashMap(); // Conflicting source functions held for next round + MultiValuedMap destHoldOn = + new HashSetValuedHashMap(); // Conflicting dest functions held for next round + MultiValuedMap sourceFormatted = + new HashSetValuedHashMap(); // Current set of potential pairs, indexed by source + MultiValuedMap destFormatted = + new HashSetValuedHashMap(); // Current set of potential pairs, indexed by dest + + for (FunctionPair pair : discoveredMatches) { // Copy putative matches into the "current" set of potential pairs + sourceFormatted.put(pair.getSourceNode(), pair); + destFormatted.put(pair.getDestNode(), pair); + } + discoveredMatches = null; // The raw match list is no longer needed beyond this point + + int keepLen = sourceFormatted.size(); + if (keepLen == 0) { + return; + } + + boolean changed = true; + double ratioThresh = .5; // Initial threshold for throwing out pairs. Counts can differ by a factor of 2 to 1. + while (changed) { // Keep going until no change (no new pairs) + monitor.checkCancelled(); + final Collection values = sourceFormatted.values(); + monitor.initialize(values.size()); + for (FunctionPair entry : values) { + monitor.checkCancelled(); + monitor.incrementProgress(1); + if (!hasConflicts(entry, sourceFormatted, destFormatted)) { // Check for conflicts in our current set + finalPairs.add(entry); // Accept immediately if no conflicts + matchedSource.add(entry.getSourceNode()); + matchedDest.add(entry.getDestNode()); + } + else { + if (!matchedSource.contains(entry.getSourceNode()) && + !matchedDest.contains(entry.getDestNode())) { + // If there is a conflict, but neither side has been matched yet, + // decide if we throw out pair by comparing count ratios to ratioThresh + + // Compute "number of children" ratio + double leftside = + Math.min((double) entry.getSourceNode().getChildren().size(), + (double) entry.getDestNode().getChildren().size()); + double rightside = + Math.max((double) entry.getSourceNode().getChildren().size(), + (double) entry.getDestNode().getChildren().size()); + double childRatio = (rightside == 0 ? 0 : leftside / rightside); // Always <= 1.0 + + // Compute byte length ratio + leftside = (double) entry.getSourceNode().getLen() / + (double) entry.getDestNode().getLen(); + double lenRatio = Math.min(leftside, 1 / leftside); // Always <= 1.0 + if (lenRatio > ratioThresh && childRatio > ratioThresh) { // Test both ratios against threshold + // Keep (don't throw out) if both ratios exceed threshold + sourceHoldOn.put(entry.getSourceNode(), entry); + destHoldOn.put(entry.getDestNode(), entry); + } + } + } + } + sourceFormatted = sourceHoldOn; // Update our "current" set of sources + destFormatted = destHoldOn; // Update our "current" set of dests + changed = (keepLen != values.size()); // Did we get any new pairs this round? + keepLen = sourceHoldOn.values().size(); + sourceHoldOn = new HashSetValuedHashMap(); + destHoldOn = new HashSetValuedHashMap(); + ratioThresh = (2 + ratioThresh) / 3; // Tighten the ratio threshold for next round + // Move closer to 1.0 threshold (counts are exactly equal) + } + if (finalPairs.isEmpty()) { + return; // found no seeds + } + Collections.sort(finalPairs, CONF_COMPARATOR); + + double curConf = finalPairs.get(0).getConfResult(); + if (curConf < confThreshold) { + Msg.warn(this, "Initial value of seed confidence too high (" + confThreshold + + ")...resetting seed confidence to " + curConf); + confThreshold = curConf; + } + int lastIndex = findIndexMatchingThreshold(finalPairs, confThreshold); // Last index that still meets threshold + for (int i = 0; i < lastIndex + 1; ++i) { + FunctionPair pair = finalPairs.get(i); + seeds.add(pair); + } + } + + private static boolean hasConflicts(FunctionPair entry, + MultiValuedMap sourceFormatted, + MultiValuedMap destFormatted) { + Collection sources = sourceFormatted.get(entry.getSourceNode()); + if (sources != null && sources.size() > 1) { + return true; + } + Collection dests = destFormatted.get(entry.getDestNode()); + if (dests != null && dests.size() > 1) { + return true; + } + return false; + } + + /** + * Generate seed matches, placing the FunctionPair into the {@link #seeds} container. + * Seeds come from a) previously accepted matches and b) the {@link #discoveredMatches} + * @param matchSet is used to identify already accepted matches + * @param useAcceptedMatchesAsSeeds is true if previously accepted matches are considered seeds + * @param monitor is the TaskMonitor + * @return true if at least one seed was identified + * @throws CancelledException if the user cancels the correlation + */ + public boolean generateSeeds(VTMatchSet matchSet, boolean useAcceptedMatchesAsSeeds, + TaskMonitor monitor) throws CancelledException { + seeds = new HashSet(); + if (useAcceptedMatchesAsSeeds) { + findAcceptedSeeds(matchSet, monitor); + } + chooseSeeds(monitor); + return !seeds.isEmpty(); + } + + /** + * Establish what neighborhood generation strategy will be used + * @param round - which round to build a strategy for + * @return an array of NeighborGenerators + */ + private NeighborGenerator[] buildNeighborGenerators(int round) { + ArrayList generatorList = new ArrayList(); + if (round == 0) { + // For first round only collect new matches from "close" relationships (i.e. parent/child) + // of the seed match. + generatorList.add(new NeighborGenerator.Children(vectorFactory, impThreshold)); + generatorList.add(new NeighborGenerator.Parents(vectorFactory, impThreshold)); + // If the format includes explicit namespace information for functions, + // use it when generating new matches. + if (useNamespaceNeighbors) { + generatorList.add( + new NamespaceNeighborhood(vectorFactory, impThreshold, sourceNodes, destNodes)); + } + } + else { + // For later rounds, also collect matches from more distant relationships (grandparent, grandchild, etc.) + generatorList.add(new NeighborGenerator.Children(vectorFactory, impThreshold)); + generatorList.add(new NeighborGenerator.Parents(vectorFactory, impThreshold)); + generatorList.add(new NeighborGenerator.GrandChildren(vectorFactory, impThreshold)); + generatorList.add(new NeighborGenerator.Siblings(vectorFactory, impThreshold)); + generatorList.add(new NeighborGenerator.Spouses(vectorFactory, impThreshold)); + generatorList.add(new NeighborGenerator.GrandParents(vectorFactory, impThreshold)); + if (useNamespaceNeighbors) { + generatorList.add( + new NamespaceNeighborhood(vectorFactory, impThreshold, sourceNodes, destNodes)); + } + } + NeighborGenerator[] res = new NeighborGenerator[generatorList.size()]; + generatorList.toArray(res); + return res; + } + + /** + * Given a set of -seeds- iteratively extend the set of matches + * Loop greedily picking the best relative match, maintaining score sorts and other bookkeeping + * @param monitor is the TaskMonitor + * @return the final list of FunctionPairs as official matches + * @throws CancelledException if the user cancels the correlation + */ + public List doMatching(TaskMonitor monitor) throws CancelledException { + matches = new LinkedList(); + + for (int round = 0; round < 2; round++) { + monitor.checkCancelled(); + NeighborGenerator[] generatorList = buildNeighborGenerators(round); + if (round == 0) { + monitor.setMessage("Matching round 1..."); + monitor.initialize(seeds.size()); + for (FunctionPair bridge : seeds) { + monitor.checkCancelled(); + monitor.incrementProgress(1); + acceptMatch(bridge); + PotentialPair impliedPair = analyze(bridge, generatorList); + if (impliedPair != null) { + implications.add(impliedPair); + } + } + seeds = null; // seeds are no longer needed, free up memory + } + else { + implications.clear(); + monitor.setMessage("Matching round 2..."); + monitor.initialize(matches.size()); + for (FunctionPair bridge : matches) { + monitor.checkCancelled(); + monitor.incrementProgress(1); + PotentialPair impliedPair = analyze(bridge, generatorList); + if (impliedPair != null) { + implications.add(impliedPair); + } + } + } + monitor.setMessage("Gathering matches for round " + (round + 1) + "..."); + int maxSize = implications.size(); + monitor.initialize(maxSize + 1); + while (true) { + monitor.checkCancelled(); + int size = implications.size(); + if (size > maxSize) { + maxSize = size; + monitor.setMaximum(maxSize + 1); + } + monitor.setProgress((maxSize - size) + 1); + if (size == 0) { + break; + } + PotentialPair bestImplied = implications.last(); + implications.remove(bestImplied); + FunctionPair bridge = + bestImplied.getSource().findEdge(bestImplied.getDestination()); + if (bridge != null) { + acceptMatch(bridge); + PotentialPair impliedPair = analyze(bridge, generatorList); + if (impliedPair != null) { + implications.add(impliedPair); + } + } + // Let pair that produced this new match select a new PotentialPair + PotentialPair impliedPair = analyze(bestImplied.getOrigin(), generatorList); + if (impliedPair != null) { + implications.add(impliedPair); + } + if (implications.isEmpty() || implications.last().getScore() < impThreshold) { + break; + } + } + } + + //Hole Patching + LinkedList matchCopy = new LinkedList(matches); + VectorCompare veccompare = new VectorCompare(); + monitor.setMessage("Patching holes..."); + monitor.initialize(matches.size()); + for (FunctionPair bridge : matchCopy) { + monitor.checkCancelled(); + monitor.incrementProgress(1); + if (bridge.getSourceNode().getParents().size() == 1 && + bridge.getDestNode().getParents().size() == 1) { + FunctionNode sp = bridge.getSourceNode().getParents().iterator().next(); + FunctionNode dp = bridge.getDestNode().getParents().iterator().next(); + if (sp.findEdge(dp) == null && !sp.isAcceptedMatch() && !dp.isAcceptedMatch()) { + double similarity = sp.getVector().compare(dp.getVector(), veccompare); + double confidence = vectorFactory.calculateSignificance(veccompare); + FunctionPair rentBridge = new FunctionPair(sp, dp, similarity, confidence); + acceptMatch(rentBridge); + } + } + } + + return matches; + } + + //Compare pairs by confidence. + private static final Comparator CONF_COMPARATOR = new Comparator() { + @Override + public int compare(FunctionPair o1, FunctionPair o2) { + return Double.compare(o2.getConfResult(), o1.getConfResult()); + } + }; + + /** + * Run through the VersionTrack match-set looking for matches between functions + * that have been formally marked as "accepted" + * @param myMatchSet is the match-set to examine + * @param monitor is the TaskMonitor + * @throws CancelledException if the user cancels the correlation + */ + private void findAcceptedSeeds(VTMatchSet myMatchSet, TaskMonitor monitor) + throws CancelledException { + monitor.setMessage("Using accepted matches as seeds..."); + VTSession session = myMatchSet.getSession(); + VTAssociationManager associationManager = session.getAssociationManager(); + int associationCount = associationManager.getAssociationCount(); + monitor.initialize(associationCount); + List associations = associationManager.getAssociations(); + Program sourceProgram = sourceNodes.getProgram(); + Program destinationProgram = destNodes.getProgram(); + + for (VTAssociation association : associations) { + monitor.checkCancelled(); + if (association.getType().equals(VTAssociationType.FUNCTION) && + association.getStatus() == VTAssociationStatus.ACCEPTED) { + + Address sourceAddress = association.getSourceAddress(); + Function sourceFunction = sourceProgram.getListing().getFunctionAt(sourceAddress); + Address destinationAddress = association.getDestinationAddress(); + Function destinationFunction = + destinationProgram.getListing().getFunctionAt(destinationAddress); + + if (sourceFunction != null && destinationFunction != null) { + FunctionNode sn = sourceNodes.get(sourceAddress); + if (sn != null) { + FunctionNode dn = destNodes.get(destinationAddress); + if (dn != null) { + FunctionPair bridge = sn.findEdge(dn); + if (bridge != null) { + seeds.add(bridge); + } + } + } + } + } + monitor.incrementProgress(1); + } + } + + /** + * Given an accepted FunctionPair and methods for generating neighborhoods, + * For each generation method, generate a source neighborhood and a dest neighborhood + * and search for pairs between the two neighborhoods with the highest confidence score. + * + * @param pair is the accepted FunctionPair + * @param generatorList is the list of neighborhood generators + * @return the highest confidence pair across all pairs of neighborhoods + */ + private PotentialPair analyze(FunctionPair pair, NeighborGenerator[] generatorList) { + FunctionNode sourceNode = pair.getSourceNode(); + FunctionNode destNode = pair.getDestNode(); + double confResult = pair.getConfResult(); + + double implicationScore = 0; + PotentialPair bestImplied = null; + + for (NeighborGenerator generator : generatorList) { + NeighborhoodPair nPair = generator.generate(sourceNode, destNode); + PotentialPair srcToDestPair = + calculateBestNeighbor(nPair.srcNeighbors, nPair.destNeighbors, confResult); + if (srcToDestPair.getScore() > implicationScore) { + implicationScore = srcToDestPair.getScore(); + bestImplied = srcToDestPair; + } + PotentialPair destToSrcPair = + calculateBestNeighbor(nPair.destNeighbors, nPair.srcNeighbors, confResult); + destToSrcPair.swap(); // PotentialPair is returned with opposite from and to nodes + if (destToSrcPair.getScore() > implicationScore) { + implicationScore = destToSrcPair.getScore(); + bestImplied = destToSrcPair; + } + } + if (bestImplied != null) { + bestImplied.setOrigin(pair); + } + return bestImplied; + } + + /** + * Among a -range- of pairs with the same score, return a pair that does not conflict with + * any other pair in the range, i.e. the source and destination of the pair or not + * involved in another pair (with the same score). + * @param potentialPairs is the (ordered) set of pairs + * @param firstIndex is the start index of the range + * @param lastIndex is the last index of the range + * @return an unconflicted pair or null if none exist + */ + private static PotentialPair unconflictedPair(ArrayList potentialPairs, + int firstIndex, int lastIndex) { + for (int i = firstIndex; i <= lastIndex; i++) { + FunctionNode myFrom = potentialPairs.get(i).getSource(); + FunctionNode myTo = potentialPairs.get(i).getDestination(); + boolean useMe = true; + for (int j = firstIndex; j <= lastIndex; j++) { // Look for conflicts in entries with same score + if (i == j) { + continue; + } + FunctionNode yourFrom = potentialPairs.get(j).getSource(); + FunctionNode yourTo = potentialPairs.get(j).getDestination(); + if (myFrom == yourFrom || myTo == yourTo) { + useMe = false; // Conflict found. Can't use this one. + break; + } + } + if (useMe) { // No conflict found + return potentialPairs.get(i); // Use this entry + } + } + return null; + } + + /** + * Adjust an original confidence score between functions -a- and -b- + * based on the likelihood of children matching and parents matching. + * @param conf is the original confidence + * @param a is one side of the function pair + * @param b is the other side + * @return the adjusted score + */ + private static double adjustConfidenceScore(double conf, FunctionNode a, FunctionNode b) { + final int childrenSize = b.getChildren().size(); + double ratio = (childrenSize == 0 ? 0 : (double) a.getChildren().size() / childrenSize); + final double kidRatio = Math.min(ratio, 1 / ratio); + final int parentsSize = b.getParents().size(); + ratio = (parentsSize == 0 ? 0 : (double) a.getParents().size() / parentsSize); + final double rentRatio = Math.min(ratio, 1 / ratio); + + ratio = (double) a.getLen() / b.getLen(); + final double lenRatio = Math.min(ratio, 1 / ratio); + return 0.25 * conf * lenRatio * (1 + kidRatio) * (1 + rentRatio); + } + + /** + * Find the first PotentialPair where there is no conflict. + * Sort the pairs based on score, and divide them into ranges of equal score. + * Look for the first PotentialPair whose source and dest are not involved with any + * other pair within an equal score range. + * @param potentialPairs is the array of pairs + * @return the first (highest scoring) unconflicted pair (or null) + */ + private static PotentialPair findFirstUnconflictedPair( + ArrayList potentialPairs) { + Collections.sort(potentialPairs); // Sort pairs based on score + int lastIndex = potentialPairs.size() - 1; + while (lastIndex >= 0) { + double score = potentialPairs.get(lastIndex).getScore(); + int firstIndex = lastIndex - 1; + while (firstIndex >= 0 && potentialPairs.get(firstIndex).getScore() >= score) { + firstIndex -= 1; + } + PotentialPair bestPair = unconflictedPair(potentialPairs, firstIndex + 1, lastIndex); + if (bestPair != null) { + return bestPair; + } + lastIndex = firstIndex; + } + + return PotentialPair.EMPTY_PAIR; // No match found. We get here in the case of conflict-only matrices. + } + + /** + * Given matching neighborhoods, look at "matrix" of scores for pairs across them. + * Return the most likely pair. + * @param aNeighbors is the first neighborhood + * @param bNeighbors is the second neighborhood + * @param confResult is the confidence score associated with the accepted match + * @return the most likely pair as a PotentialPair + */ + private PotentialPair calculateBestNeighbor(Set aNeighbors, + Set bNeighbors, double confResult) { + ArrayList potentialPairs = new ArrayList(); + PotentialPair bestPair = PotentialPair.EMPTY_PAIR; + int bestCount = 0; // Number of pairs with the same (currently) best score + + // CRITICAL LOOP + for (FunctionNode relative : aNeighbors) { // For every function in the source neighborhood + if (relative.isAcceptedMatch()) { + continue; + } + double bestAdjustedScore = 0; // Best score you're seeing for just this relative. + double relSum = 0; // Sum of relative's scores for associates...for normalizing. + double bestOriginalScore = 0; // So that we can recover the entry without computation. + FunctionNode bestRelAssoc = null; // The highest scoring associate + // CRITICAL INNER LOOP + Iterator> iter = relative.getAssociateIterator(); + while (iter.hasNext()) { // Run through every putative match to -relative- + Entry entry = iter.next(); + final FunctionNode associate = entry.getKey(); + final double value = entry.getValue().getConfResult(); + if (bNeighbors.contains(associate)) { // Does the dest side of the match lie in dest neighborhood + double entryAdjusted = adjustConfidenceScore(value, relative, associate); + relSum += entryAdjusted; // Keep track of score sum for normalization + if (entryAdjusted >= bestAdjustedScore) { // Keep track of highest scoring pair + bestAdjustedScore = entryAdjusted; + bestRelAssoc = associate; + bestOriginalScore = value; + } + } + } + + if (relSum > 0) { + // Compute a final score that takes into account the dimensions of the neighborhoods + // and scores of other potential pairs across the neighborhoods + double tempMax = bNeighbors.size() * (bestOriginalScore + confResult) * + bestAdjustedScore / relSum; + + PotentialPair newPair = new PotentialPair(relative, bestRelAssoc, tempMax); + potentialPairs.add(newPair); + if (tempMax > bestPair.getScore()) { // We have seen a new maximum. + bestPair = newPair; // Keep track of the new best + bestCount = 1; // Restart the counter + } + else if (tempMax == bestPair.getScore()) { // A tie score with the current best + bestCount += 1; + } + + } + } + + if (bestCount == 0 || bestPair.getScore() == 0) { + return PotentialPair.EMPTY_PAIR; // The default null object passed for nothing found. + } + + if (bestCount == 1) { // There is a unique best entry. Use it. + return bestPair; + } + + return findFirstUnconflictedPair(potentialPairs); // The best pair is a tie, we need to go deeper into the list + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BinningSystem.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BinningSystem.java new file mode 100755 index 0000000000..7f3716d9e1 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/BinningSystem.java @@ -0,0 +1,119 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import java.util.*; + +import generic.lsh.*; +import generic.lsh.vector.HashEntry; +import ghidra.util.exception.CancelledException; +import ghidra.util.task.TaskMonitor; + +/** + * Container for FunctionNodes so that nodes that are "near" each other + * (meaning the nodes' feature vectors have high cosine-similarity) + * can be discovered. As nodes are added, they are distributed across + * bins, where similar nodes tend to be placed into the same bins. + */ +class BinningSystem { + private final int L; // Number of distinct binnings + + private int[][] partitionIdentities; + private TreeMap>[] binSys; + + /** + * Construct a container that holds the FunctionNodes. If model is not null, then the FunctionNodes will be indexed + * @param model is the particular configuration model to use for this + */ + @SuppressWarnings("unchecked") + public BinningSystem(LSHMemoryModel model) { + int k = model.getK(); // k = #of hyperplanes comprising the each binning. + L = KandL.memoryModelToL(model); + this.partitionIdentities = new int[L][]; + this.binSys = new TreeMap[L]; // A system of L binnings. + Random random = new Random(23); + for (int ii = 0; ii < L; ++ii) { + this.partitionIdentities[ii] = new int[k]; + for (int jj = 0; jj < k; ++jj) { + this.partitionIdentities[ii][jj] = random.nextInt(); + } + this.binSys[ii] = new TreeMap>(); + } + } + + /** + * Add a list of {@link FunctionNode} objects into the bins + * @param iter is an iterator over the raw FunctionNodes to add + * @param monitor is the TaskMonitor + * @throws CancelledException for user cancellation of the correlator + */ + public void add(Iterator iter, TaskMonitor monitor) throws CancelledException { + + while (iter.hasNext()) { + FunctionNode node = iter.next(); + monitor.checkCancelled(); + monitor.incrementProgress(1); + if (node.getVector() == null) { + continue; + } + int[] features = getBinIds(node); + for (int ii = 0; ii < features.length; ++ii) { + TreeSet list = binSys[ii].get(features[ii]); + if (list == null) { + list = new TreeSet(); + binSys[ii].put(features[ii], list); + } + list.add(node); + } + } + } + + /** + * Returns the union of all the bins containing the exemplar FunctionNode. + * These nodes are likely to similar to the exemplar, but need secondary testing. + * @param node is the exemplar + * @return a set of FunctionNodes + */ + public Set lookup(FunctionNode node) { + TreeSet result = new TreeSet(); + int[] features = getBinIds(node); + for (int ii = 0; ii < features.length; ++ii) { + TreeSet list = binSys[ii].get(features[ii]); + if (list != null) { + result.addAll(list); + } + } + return result; + } + + /** + * Given a node, calculate the binId for each binning in this system + * @param node is the FunctionNode to label + * @return an array of ids + */ + private int[] getBinIds(FunctionNode node) { + if (node.getVector() == null) { + return null; + } + int[] result = new int[L]; + HashEntry[] entries = node.getVector().getEntries(); + for (int ii = 0; ii < L; ++ii) { + int hash = Partition.hash(partitionIdentities[ii], entries); + result[ii] = hash; + } + return result; + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionNode.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionNode.java new file mode 100755 index 0000000000..f9b8e8c7a2 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionNode.java @@ -0,0 +1,200 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import java.util.*; +import java.util.Map.Entry; + +import generic.lsh.vector.LSHVector; +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.Function; + +/** + * Information about a single function the correlator is attempting to match + */ +public class FunctionNode implements Comparable { + + private final Address addr; // Address of the function represented, also unique identifier + private final String name; // Name of the function this node represents. + private final LSHVector vec; // Feature vector + private ArrayList
callAddresses; // Addresses of functions this node calls. + private final Set children; // Who do I call in the call graph? + private final Set parents; // Who calls me in the call graph? + private Map associates; // Potential matches on the other side? And what's our conf? + private final int len; // Number of addresses in the body of this function + private boolean acceptedMatch; // Has this node been formally matched with something + + /** + * Allocate a container for FunctionNodes as needed by the NeighborGenerators. These are generally small sets + * where we need to check containment constantly. + * @return the container + */ + public static Set neigborhoodAllocate() { + return new HashSet(); + } + + public FunctionNode(Function function, LSHVector vector, ArrayList
callAddresses) { + this.addr = function.getEntryPoint(); + this.name = function.getName(); + this.vec = vector; + this.callAddresses = callAddresses; //It will take a second pass through the data to figure out how the call graph fits together. + this.associates = new HashMap(); + this.children = neigborhoodAllocate(); + this.parents = neigborhoodAllocate(); + int val = (int) function.getBody().getNumAddresses(); + this.len = (val == 0) ? 1 : val; // Guarantee a non-zero length + this.acceptedMatch = false; + } + + @Override + public int hashCode() { + return ((addr == null) ? 0 : addr.hashCode()); + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null) { + return false; + } + if (getClass() != obj.getClass()) { + return false; + } + FunctionNode other = (FunctionNode) obj; + if (addr == null) { + if (other.addr != null) { + return false; + } + } + else if (!addr.equals(other.addr)) { + return false; + } + return true; + } + + @Override + public int compareTo(FunctionNode other) { + return addr.compareTo(other.addr); // Compare by address + } + + @Override + public String toString() { + return name; + } + + /** + * @return the Address of the entry point of the Function represented by this node + */ + public Address getAddress() { + return addr; + } + + /** + * @return the feature vector associated with this node (function) + */ + public LSHVector getVector() { + return vec; + } + + /** + * Grab the raw call addresses, releasing the memory in the process + * @return the list of addresses + */ + public List
releaseCallAddresses() { + List
res = callAddresses; + callAddresses = null; // Release our reference to addresses + return res; + } + + /** + * @return the set of functions (FunctionNodes) called by this function + */ + public Set getChildren() { + return children; + } + + /** + * @return the set of functions (FunctionNodes) that call this function + */ + public Set getParents() { + return parents; + } + + /** + * Add a (potential) match for this node. The match + * is stored with a FunctionPair object holding similarity information + * @param other is the potentially matching FunctionNode + * @param pair is the FunctionPair describing the similarity + */ + public void addAssociate(FunctionNode other, FunctionPair pair) { + associates.put(other, pair); + } + + /** + * Remove what was previously considered a potential match. + * @param other is the matching FunctionNode + */ + public void removeAssociate(FunctionNode other) { + associates.remove(other); + } + + /** + * Clear all potential matches. + */ + public void clearAssociates() { + associates.clear(); + } + + /** + * @return an iterator over all potential matches for this node + */ + public Iterator> getAssociateIterator() { + return associates.entrySet().iterator(); + } + + /** + * If -other- is a potential match, return the FunctionPair describing the similarity + * @param other is the possible potential match + * @return the FunctionPair describing the match or null, if -other- is not a potential match + */ + public FunctionPair findEdge(FunctionNode other) { + return associates.get(other); + } + + /** + * @return the number of addresses in the function body represented by this node + */ + public int getLen() { + return len; + } + + /** + * @return true if this node has been formally matched by the correlator + */ + public boolean isAcceptedMatch() { + return acceptedMatch; + } + + /** + * Mark that this node has been matched (not matched) by the correlator + * @param used is true if this node has been matched + */ + public void setAcceptedMatch(boolean used) { + this.acceptedMatch = used; + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionNodeContainer.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionNodeContainer.java new file mode 100755 index 0000000000..e2ddc5df62 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionNodeContainer.java @@ -0,0 +1,101 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import java.util.*; + +import ghidra.program.model.address.Address; +import ghidra.program.model.listing.*; + +/** + * Container of FunctionNodes corresponding to functions in a single Program + */ +public class FunctionNodeContainer { + private Program program; // Program containing all the functions + private Map addrToNode; // Map from Address to FunctionNode representing the function + + public FunctionNodeContainer(Program program, List nodeList) { + this.program = program; + addrToNode = new TreeMap(); + for (FunctionNode node : nodeList) { + addrToNode.put(node.getAddress(), node); + } + generateCallGraph(); + } + + public Program getProgram() { + return program; + } + + /** + * Get the FunctionNode associated with a specific address + * @param addr the Address to search for + * @return the corresponding FunctionNode (or null if addr maps to nothing) + */ + public FunctionNode get(Address addr) { + return addrToNode.get(addr); + } + + /** + * @return the number of FunctionNodes held in this container + */ + public int size() { + return addrToNode.size(); + } + + /** + * @return an iterator over all FunctionNodes in this container, in address order + */ + public Iterator iterator() { + return addrToNode.values().iterator(); + } + + /** + * Generate program call-graph in terms of FunctionNodes + * Uses the call address attached to each raw FunctionNode + * Once the xrefs are built, the original call address arrays are released + */ + private void generateCallGraph() { + FunctionManager mgr = program.getFunctionManager(); + for (FunctionNode node : addrToNode.values()) { //Addresses are associated to nodes. + if (node != null) { + List
callAddresses = node.releaseCallAddresses(); + for (Address addr : callAddresses) { + FunctionNode kid; + for (;;) { + kid = addrToNode.get(addr); //These nodes are the vertices in the call graph. + if (kid != null) { + break; + } + Function f = mgr.getFunctionAt(addr); // If addr does not link to a node, it is most likely a thunk + if (f == null) { + break; + } + if (!f.isThunk()) { + break; + } + addr = f.getThunkedFunction(false).getEntryPoint(); // Replace with address of thunked function + } + if (kid != null) { + node.getChildren().add(kid); + kid.getParents().add(node); + } + } + } + } + return; + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionPair.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionPair.java new file mode 100755 index 0000000000..1410ddbdcd --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/FunctionPair.java @@ -0,0 +1,133 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import ghidra.feature.vt.api.main.*; + +/** + * A possible match between source and destination. + */ +public class FunctionPair { + + private FunctionNode sourceNode; // Function from the source program + private FunctionNode destNode; // Function from the destination program + private double simResult; // Similarity of the pair (0.0 to 1.0) + private double confResult; // Confidence score of the pair + + /** + * Constructor + * @param source the source function + * @param dest the destination function + * @param simRes the computed similarity score + * @param confRes the computed confidence score + */ + public FunctionPair(FunctionNode source, FunctionNode dest, double simRes, double confRes) { + this.sourceNode = source; + this.destNode = dest; + this.simResult = simRes; + this.confResult = confRes; + } + + @Override + public int hashCode() { + final int prime = 31; + int result = 1; + result = prime * result + ((destNode == null) ? 0 : destNode.hashCode()); + result = prime * result + ((sourceNode == null) ? 0 : sourceNode.hashCode()); + return result; + } + + @Override + public boolean equals(Object obj) { + if (this == obj) { + return true; + } + if (obj == null) { + return false; + } + if (getClass() != obj.getClass()) { + return false; + } + FunctionPair other = (FunctionPair) obj; + if (destNode == null) { + if (other.destNode != null) { + return false; + } + } + else if (!destNode.equals(other.destNode)) { + return false; + } + if (sourceNode == null) { + if (other.sourceNode != null) { + return false; + } + } + else if (!sourceNode.equals(other.sourceNode)) { + return false; + } + return true; + } + + /** + * Compute the formal Version Tracking match record corresponding to this pair + * @param matchSet is the match set the record should be added to + * @return the match record + */ + public VTMatchInfo getMatch(VTMatchSet matchSet) { + VTMatchInfo result = new VTMatchInfo(matchSet); + result.setSimilarityScore(new VTScore(simResult)); + result.setConfidenceScore(new VTScore(confResult)); + result.setAssociationType(VTAssociationType.FUNCTION); + result.setSourceAddress(sourceNode.getAddress()); + result.setDestinationAddress(destNode.getAddress()); + result.setSourceLength(sourceNode.getLen()); + result.setDestinationLength(destNode.getLen()); + return result; + } + + @Override + public String toString() { + return sourceNode.toString() + "," + destNode.toString(); + } + + /** + * @return info about the source function + */ + public FunctionNode getSourceNode() { + return sourceNode; + } + + /** + * @return info about the destination function + */ + public FunctionNode getDestNode() { + return destNode; + } + + /** + * @return the similarity score of the pair + */ + public double getSimResult() { + return simResult; + } + + /** + * @return the confidence score of the pair + */ + public double getConfResult() { + return confResult; + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/NamespaceNeighborhood.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/NamespaceNeighborhood.java new file mode 100755 index 0000000000..13d393b123 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/NamespaceNeighborhood.java @@ -0,0 +1,136 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import java.util.Set; +import java.util.TreeMap; + +import generic.lsh.vector.LSHVectorFactory; +import ghidra.program.model.listing.Function; +import ghidra.program.model.symbol.*; + +/** + * A neighborhood generator that, for a given function, generates all functions + * in the same namespace. For efficiency, it caches the namespace sets it generates. + */ +public class NamespaceNeighborhood extends NeighborGenerator { + + private FunctionNodeContainer sourceNodes; // Reference to global set of source functions + private FunctionNodeContainer destNodes; // Reference to global set of destination functions + private TreeMap> sourceSets; // Map from namespace ID to matching set of source functions + private TreeMap> destSets; // Map from namespace ID to matching set of dest functions + private TreeMap namespacePair; // Map from pair of namespace IDs to pair of namespace sets + private PairLabel cacheKey; // internal key for quick lookups into namespacePair map + + private static class PairLabel implements Comparable { + public Long srcLabel; + public Long destLabel; + + @Override + public int compareTo(PairLabel o) { + int srcCmp = Long.compare(srcLabel.longValue(), o.srcLabel.longValue()); + if (srcCmp != 0) { + return srcCmp; + } + return Long.compare(destLabel.longValue(), o.destLabel.longValue()); + } + } + + public NamespaceNeighborhood(LSHVectorFactory vectorFactory, double impThreshold, + FunctionNodeContainer sourceNodes, FunctionNodeContainer destNodes) { + super(vectorFactory, impThreshold); + this.sourceNodes = sourceNodes; + this.destNodes = destNodes; + sourceSets = new TreeMap>(); + destSets = new TreeMap>(); + namespacePair = new TreeMap(); + cacheKey = new PairLabel(); + } + + private Namespace getNamespace(FunctionNode root, FunctionNodeContainer container) { + Function function = + container.getProgram().getFunctionManager().getFunctionAt(root.getAddress()); + if (function == null) { + return null; + } + Namespace namespace = function.getParentNamespace(); + return namespace; + } + + private Set buildNeighborhood(Namespace namespace, Long namespaceKey, + FunctionNodeContainer container, TreeMap> sets) { + Set resultSet = sets.get(namespaceKey); + if (resultSet == null) { + resultSet = FunctionNode.neigborhoodAllocate(); + SymbolTable symbolTable = container.getProgram().getSymbolTable(); + SymbolIterator iter = symbolTable.getSymbols(namespace); + while (iter.hasNext()) { + Symbol sym = iter.next(); + if (sym.getSymbolType() != SymbolType.FUNCTION) { + continue; + } + FunctionNode node = container.get(sym.getAddress()); + if (node != null) { + resultSet.add(node); + } + } + sets.put(namespaceKey, resultSet); + } + return resultSet; + } + + private NeighborhoodPair findPair(Long srcKey, Long destKey) { + cacheKey.srcLabel = srcKey; + cacheKey.destLabel = destKey; + return namespacePair.get(cacheKey); + } + + private void cachePair(Long srcKey, Long destKey, NeighborhoodPair pair) { + PairLabel newLabel = new PairLabel(); + newLabel.srcLabel = srcKey; + newLabel.destLabel = destKey; + namespacePair.put(newLabel, pair); + } + + @Override + public NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot) { + Namespace srcNamespace = getNamespace(srcRoot, sourceNodes); + Namespace destNamespace = getNamespace(destRoot, destNodes); + if (srcNamespace == null || destNamespace == null) { + NeighborhoodPair pair = new NeighborhoodPair(); + pair.srcNeighbors = FunctionNode.neigborhoodAllocate(); + pair.destNeighbors = FunctionNode.neigborhoodAllocate(); + return pair; // Empty pair + } + Long srcNamespaceKey = srcNamespace.getID(); + Long destNamespaceKey = destNamespace.getID(); + NeighborhoodPair pair = findPair(srcNamespaceKey, destNamespaceKey); + if (pair == null) { + pair = new NeighborhoodPair(); + pair.srcNeighbors = + buildNeighborhood(srcNamespace, srcNamespaceKey, sourceNodes, sourceSets); + pair.destNeighbors = + buildNeighborhood(destNamespace, destNamespaceKey, destNodes, destSets); + cachePair(srcNamespaceKey, destNamespaceKey, pair); + } + if (!pair.isFilledOut) { + if (fillOutPairs(pair, 10000)) { + pair.isFilledOut = true; + } + } + return pair; + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/NeighborGenerator.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/NeighborGenerator.java new file mode 100755 index 0000000000..dcea465a22 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/NeighborGenerator.java @@ -0,0 +1,287 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import java.util.ArrayList; +import java.util.Set; + +import generic.lsh.vector.*; + +/** + * Class(es) for constructing a "neighborhood" of functions around a function + * that we know has a match. Comparing across neighborhoods provides a large + * cut-down in both search time and uncertainty when trying to find additional matches. + */ +public abstract class NeighborGenerator { + + public static final int RELATIVE_COMPARES = 25; // Maximum number of extra compares between "relative" sets + private double impThreshold; // Confidence threshold for extending to additional matches + private LSHVectorFactory vectorFactory; + + public static class NeighborhoodPair { + public Set srcNeighbors; + public Set destNeighbors; + public boolean isFilledOut = false; + } + + public NeighborGenerator(LSHVectorFactory vectorFactory, double impThreshold) { + this.vectorFactory = vectorFactory; + this.impThreshold = impThreshold; + } + + /** + * Given roots from the source program and the destination program, + * generate a neighborhood of functions related to each root. + * @param srcRoot is the root from the source program + * @param destRoot is the root from the destination program + * @return a pair of "neighborhoods" as a set of FunctionNodes + */ + public abstract NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot); + + /** + * Do the feature vector comparison of every source to every destination and create + * new putative matches (associates) if the comparison score exceeds {@link #impThreshold} + * @param unmatchedSource is the list of sources + * @param unmatchedDest is the list of destinations + */ + private void searchForNewMatches(ArrayList unmatchedSource, + ArrayList unmatchedDest) { + VectorCompare veccompare = new VectorCompare(); + for (FunctionNode src : unmatchedSource) { + LSHVector srcvec = src.getVector(); + for (FunctionNode dst : unmatchedDest) { + if (src.findEdge(dst) != null) { + continue; // This pair has already been compared + } + // Feature vector computations + double similarity = srcvec.compare(dst.getVector(), veccompare); + double confidence = vectorFactory.calculateSignificance(veccompare); + if (confidence < impThreshold) { + continue; + } + FunctionPair newPair = new FunctionPair(src, dst, similarity, confidence); + src.addAssociate(dst, newPair); + dst.addAssociate(src, newPair); + } + } + } + + /** + * If nodes haven't been compared before, compare them and add an associate if it passes threshold + * @param pair is the two sets of nodes that we are comparing between + * @param maxCompares is the maximum number of comparisons to perform + * @return true is comparisons were actually performed + */ + protected boolean fillOutPairs(NeighborhoodPair pair, int maxCompares) { + ArrayList unmatchedSource = new ArrayList(); + ArrayList unmatchedDest = null; + + for (FunctionNode src : pair.srcNeighbors) { + if (src.isAcceptedMatch()) { + continue; + } + if (src.getVector() == null) { + continue; + } + unmatchedSource.add(src); + } + + if (unmatchedSource.isEmpty()) { + return false; + } + if (unmatchedSource.size() > maxCompares) { + return false; + } + unmatchedDest = new ArrayList(); + + for (FunctionNode dst : pair.destNeighbors) { + if (dst.isAcceptedMatch()) { + continue; + } + if (dst.getVector() == null) { + continue; + } + unmatchedDest.add(dst); + } + if (unmatchedDest.isEmpty()) { + return false; + } + if (unmatchedSource.size() * unmatchedDest.size() > maxCompares) { + return false; + } + + searchForNewMatches(unmatchedSource, unmatchedDest); + return true; + } + + /** + * Parents of -root- + */ + public static class Parents extends NeighborGenerator { + + public Parents(LSHVectorFactory vectorFactory, double impThreshold) { + super(vectorFactory, impThreshold); + } + + @Override + public NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot) { + NeighborhoodPair pair = new NeighborhoodPair(); + pair.srcNeighbors = srcRoot.getParents(); + pair.destNeighbors = destRoot.getParents(); + fillOutPairs(pair, RELATIVE_COMPARES); + return pair; + } + } + + /** + * Children of -root- + */ + public static class Children extends NeighborGenerator { + + public Children(LSHVectorFactory vectorFactory, double impThreshold) { + super(vectorFactory, impThreshold); + } + + @Override + public NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot) { + NeighborhoodPair pair = new NeighborhoodPair(); + pair.srcNeighbors = srcRoot.getChildren(); + pair.destNeighbors = destRoot.getChildren(); + fillOutPairs(pair, RELATIVE_COMPARES); + return pair; + } + } + + /** + * Grand parents of -root- + */ + public static class GrandParents extends NeighborGenerator { + + public GrandParents(LSHVectorFactory vectorFactory, double impThreshold) { + super(vectorFactory, impThreshold); + } + + @Override + public NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot) { + NeighborhoodPair pair = new NeighborhoodPair(); + Set tempRels = srcRoot.getParents(); + pair.srcNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.srcNeighbors.addAll(rel.getParents()); + } + pair.srcNeighbors.remove(srcRoot); + + tempRels = destRoot.getParents(); + pair.destNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.destNeighbors.addAll(rel.getParents()); + } + pair.destNeighbors.remove(destRoot); + fillOutPairs(pair, RELATIVE_COMPARES); + return pair; + } + } + + /** + * Grandchildren of -root- + */ + public static class GrandChildren extends NeighborGenerator { + + public GrandChildren(LSHVectorFactory vectorFactory, double impThreshold) { + super(vectorFactory, impThreshold); + } + + @Override + public NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot) { + NeighborhoodPair pair = new NeighborhoodPair(); + Set tempRels = srcRoot.getChildren(); + pair.srcNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.srcNeighbors.addAll(rel.getChildren()); + } + pair.srcNeighbors.remove(srcRoot); + + tempRels = destRoot.getChildren(); + pair.destNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.destNeighbors.addAll(rel.getChildren()); + } + pair.destNeighbors.remove(destRoot); + fillOutPairs(pair, RELATIVE_COMPARES); + return pair; + } + } + + /** + * Functions that share a parent with -root- + */ + public static class Siblings extends NeighborGenerator { + + public Siblings(LSHVectorFactory vectorFactory, double impThreshold) { + super(vectorFactory, impThreshold); + } + + @Override + public NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot) { + NeighborhoodPair pair = new NeighborhoodPair(); + Set tempRels = srcRoot.getParents(); + pair.srcNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.srcNeighbors.addAll(rel.getChildren()); + } + pair.srcNeighbors.remove(srcRoot); + + tempRels = destRoot.getParents(); + pair.destNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.destNeighbors.addAll(rel.getChildren()); + } + pair.destNeighbors.remove(destRoot); + fillOutPairs(pair, RELATIVE_COMPARES); + return pair; + } + } + + /** + * Functions that share a child with -root- + */ + public static class Spouses extends NeighborGenerator { + + public Spouses(LSHVectorFactory vectorFactory, double impThreshold) { + super(vectorFactory, impThreshold); + } + + @Override + public NeighborhoodPair generate(FunctionNode srcRoot, FunctionNode destRoot) { + NeighborhoodPair pair = new NeighborhoodPair(); + Set tempRels = srcRoot.getChildren(); + pair.srcNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.srcNeighbors.addAll(rel.getParents()); + } + pair.srcNeighbors.remove(srcRoot); + + tempRels = destRoot.getChildren(); + pair.destNeighbors = FunctionNode.neigborhoodAllocate(); + for (FunctionNode rel : tempRels) { + pair.destNeighbors.addAll(rel.getParents()); + } + pair.destNeighbors.remove(destRoot); + fillOutPairs(pair, RELATIVE_COMPARES); + return pair; + } + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/PotentialPair.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/PotentialPair.java new file mode 100755 index 0000000000..aab01c1118 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/api/PotentialPair.java @@ -0,0 +1,66 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +/** + * Given a matching FunctionPair, this object represents a different + * potential match taken from neighborhoods of the match endpoints. + */ +public class PotentialPair implements Comparable { + private FunctionPair originBridge; // Accepted match that induced this potential match + private FunctionNode fromNode; // Source node of potential match + private FunctionNode toNode; // Destination node of potential match + private double score; // implication score associated with potential match + + public static final PotentialPair EMPTY_PAIR = new PotentialPair(null, null, 0.0); + + public PotentialPair(FunctionNode src, FunctionNode dest, double sc) { + fromNode = src; + toNode = dest; + score = sc; + } + + public double getScore() { + return score; + } + + public FunctionNode getSource() { + return fromNode; + } + + public FunctionNode getDestination() { + return toNode; + } + + public FunctionPair getOrigin() { + return originBridge; + } + + public void setOrigin(FunctionPair pair) { + originBridge = pair; + } + + public void swap() { + FunctionNode tmp = fromNode; + fromNode = toNode; + toNode = tmp; + } + + @Override + public int compareTo(PotentialPair o) { + return Double.compare(score, o.score); + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/gui/validator/DecompilerParameterIDVTPreconditionValidator.java b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/gui/validator/DecompilerParameterIDVTPreconditionValidator.java new file mode 100755 index 0000000000..c61c1357a8 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/main/java/ghidra/feature/vt/gui/validator/DecompilerParameterIDVTPreconditionValidator.java @@ -0,0 +1,35 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.gui.validator; + +import ghidra.app.plugin.core.analysis.validator.PostAnalysisValidator; +import ghidra.app.plugin.core.decompiler.validator.DecompilerParameterIDValidator; +import ghidra.feature.vt.api.main.VTSession; +import ghidra.program.model.listing.Program; + +public class DecompilerParameterIDVTPreconditionValidator extends + VTPostAnalysisPreconditionValidatorAdaptor { + + public DecompilerParameterIDVTPreconditionValidator(Program sourceProgram, + Program destinationProgram, VTSession existingResults) { + super(sourceProgram, destinationProgram, existingResults); + } + + @Override + protected PostAnalysisValidator createPostAnalysisPreconditionValidator(Program program) { + return new DecompilerParameterIDValidator(program); + } +} diff --git a/Ghidra/Features/VersionTrackingBSim/src/test/java/ghidra/feature/vt/api/BSimSelfSimilarCorrelatorTest.java b/Ghidra/Features/VersionTrackingBSim/src/test/java/ghidra/feature/vt/api/BSimSelfSimilarCorrelatorTest.java new file mode 100755 index 0000000000..0d427d15c2 --- /dev/null +++ b/Ghidra/Features/VersionTrackingBSim/src/test/java/ghidra/feature/vt/api/BSimSelfSimilarCorrelatorTest.java @@ -0,0 +1,50 @@ +/* ### + * IP: GHIDRA + * EXCLUDE: YES + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.feature.vt.api; + +import ghidra.program.model.address.Address; +import ghidra.program.model.address.AddressFactory; +import ghidra.program.model.address.AddressSet; +import ghidra.program.model.address.AddressSetView; +import ghidra.program.model.address.AddressSpace; + +import org.junit.Test; + +public class BSimSelfSimilarCorrelatorTest extends AbstractSelfSimilarCorrelatorTest { + public BSimSelfSimilarCorrelatorTest( ) { + super(); + } + +@Test + public void testFlow() throws Exception { + exerciseFunctionsForFactory(new BSimProgramCorrelatorFactory(), + // with default settings these three functions won't get matched + getSourceMinus(0x010031ee, 0x01003ac0, 0x01004c1d)); + } + + private AddressSetView getSourceMinus(long... addresses) { + AddressFactory addressFactory = sourceProgram.getAddressFactory(); + AddressSpace addressSpace = addressFactory.getDefaultAddressSpace(); + AddressSet set = + new AddressSet(sourceProgram.getMemory().getInitializedAddressSet()); + for (long l : addresses) { + Address address = addressSpace.getAddress(l); + set = set.subtract(new AddressSet(address, address)); + } + return set; + } +} diff --git a/Ghidra/Framework/Generic/src/main/java/ghidra/util/datastruct/Counter.java b/Ghidra/Framework/Generic/src/main/java/ghidra/util/datastruct/Counter.java index c14d210046..275b73c0c3 100644 --- a/Ghidra/Framework/Generic/src/main/java/ghidra/util/datastruct/Counter.java +++ b/Ghidra/Framework/Generic/src/main/java/ghidra/util/datastruct/Counter.java @@ -15,15 +15,33 @@ */ package ghidra.util.datastruct; +import org.apache.commons.lang3.mutable.MutableInt; + /** * Simple class used to avoid immutable objects and autoboxing when storing changing integer * primitives in a collection. */ -public class Counter { - public int count; +public class Counter extends MutableInt { + /** + * Construct a new counter with an initial value of 0. + */ + public Counter() { + super(0); + } - @Override - public String toString() { - return Integer.toString(count); + /** + * Construct a new Counter with the given initial value. + * @param value the initial value + */ + public Counter(int value) { + super(value); + } + + /** + * Returns the value of this counter. + * @return the value of this counter + */ + public int count() { + return intValue(); } } diff --git a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/data/DataTypeUtilities.java b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/data/DataTypeUtilities.java index 91f137e655..32db5b8ebd 100644 --- a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/data/DataTypeUtilities.java +++ b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/data/DataTypeUtilities.java @@ -300,8 +300,9 @@ public class DataTypeUtilities { * Get the name of a data type with all conflict naming patterns removed. * * @param dataType data type - * @param includeCategoryPath if true the category path will be included with its - * @return name with without conflict patterns + * @param includeCategoryPath if true the category path will be included with the + * returned name (e.g., /mypath/mydt) + * @return name with optional category path included */ public static String getNameWithoutConflict(DataType dataType, boolean includeCategoryPath) { String name = includeCategoryPath ? dataType.getPathName() : dataType.getName(); @@ -422,8 +423,7 @@ public class DataTypeUtilities { } categoryPathParts.add(ns.getName()); } - return categoryPathParts.isEmpty() - ? baseCategory + return categoryPathParts.isEmpty() ? baseCategory : new CategoryPath(baseCategory, categoryPathParts); } @@ -439,7 +439,8 @@ public class DataTypeUtilities { * @param classNamespace class namespace * @return existing structure which resides within matching category. */ - public static Structure findExistingClassStruct(DataTypeManager dataTypeManager, GhidraClass classNamespace) { + public static Structure findExistingClassStruct(DataTypeManager dataTypeManager, + GhidraClass classNamespace) { Structure dt = findPreferredDataType(dataTypeManager, classNamespace, classNamespace.getName(), Structure.class, true); @@ -468,8 +469,7 @@ public class DataTypeUtilities { public static T findDataType(DataTypeManager dataTypeManager, Namespace namespace, String dtName, Class classConstraint) { - T dt = - findPreferredDataType(dataTypeManager, namespace, dtName, classConstraint, false); + T dt = findPreferredDataType(dataTypeManager, namespace, dtName, classConstraint, false); if (dt != null) { return dt; } @@ -493,8 +493,7 @@ public class DataTypeUtilities { * @return best matching data type */ public static T findNamespaceQualifiedDataType( - DataTypeManager dataTypeManager, - String dtNameWithNamespace, Class classConstraint) { + DataTypeManager dataTypeManager, String dtNameWithNamespace, Class classConstraint) { List pathList = SymbolPathParser.parse(dtNameWithNamespace); int nameIndex = pathList.size() - 1; @@ -586,7 +585,8 @@ public class DataTypeUtilities { String namespacePath) { if (namespacePath.length() == 0) { // root or unspecified namespace - prefer root category - return categoryPath.isRoot() ? CategoryMatchType.PREFERRED : CategoryMatchType.SECONDARY; + return categoryPath.isRoot() ? CategoryMatchType.PREFERRED + : CategoryMatchType.SECONDARY; } String path = categoryPath.getPath(); return path.endsWith(namespacePath) ? CategoryMatchType.PREFERRED : CategoryMatchType.NONE; @@ -610,7 +610,8 @@ public class DataTypeUtilities { String[] namespacePaths, boolean parentNamespacePreferred) { if (namespacePaths == null) { // root or unspecified namespace - prefer root category - return categoryPath.isRoot() ? CategoryMatchType.PREFERRED : CategoryMatchType.SECONDARY; + return categoryPath.isRoot() ? CategoryMatchType.PREFERRED + : CategoryMatchType.SECONDARY; } String path = categoryPath.getPath(); @@ -757,7 +758,7 @@ public class DataTypeUtilities { cmp = catPath1.compareTo(catPath2); } return cmp; - }; + }; /** * Perform a namespace qualified datatype search. diff --git a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/function/FunctionTagManagerDB.java b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/function/FunctionTagManagerDB.java index eaebfd28f5..4819e85f3e 100644 --- a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/function/FunctionTagManagerDB.java +++ b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/database/function/FunctionTagManagerDB.java @@ -214,13 +214,13 @@ public class FunctionTagManagerDB implements FunctionTagManager, ErrorHandler { private void incrementCountCache(FunctionTag tag) { if (tagCountCache != null) { - tagCountCache.get(tag).count++; + tagCountCache.get(tag).increment(); } } private void decrementCountCache(FunctionTag tag) { if (tagCountCache != null) { - tagCountCache.get(tag).count--; + tagCountCache.get(tag).decrement(); } } @@ -383,8 +383,8 @@ public class FunctionTagManagerDB implements FunctionTagManager, ErrorHandler { while (functionRecords.hasNext()) { DBRecord mappingRecord = functionRecords.next(); - DBRecord tagRecord = functionTagAdapter.getRecord( - mappingRecord.getLongValue(FunctionTagMappingAdapter.TAG_ID_COL)); + DBRecord tagRecord = functionTagAdapter + .getRecord(mappingRecord.getLongValue(FunctionTagMappingAdapter.TAG_ID_COL)); tags.add(getFunctionTagFromCache(tagRecord)); } return tags; @@ -403,7 +403,7 @@ public class FunctionTagManagerDB implements FunctionTagManager, ErrorHandler { buildTagCountCache(); } Counter counter = tagCountCache.get(tag); - return counter.count; + return counter.count(); } catch (IOException e) { dbError(e); @@ -421,7 +421,7 @@ public class FunctionTagManagerDB implements FunctionTagManager, ErrorHandler { DBRecord mappingRecord = records.next(); long tagId = mappingRecord.getLongValue(FunctionTagMappingAdapter.TAG_ID_COL); FunctionTag tag = getFunctionTag(tagId); - map.get(tag).count++; + map.get(tag).increment(); } tagCountCache = map; } diff --git a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/DataTypeConflictHandler.java b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/DataTypeConflictHandler.java index d255491289..7ac6c6f3b7 100644 --- a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/DataTypeConflictHandler.java +++ b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/DataTypeConflictHandler.java @@ -15,6 +15,16 @@ */ package ghidra.program.model.data; +/** + * {@link DataTypeConflictHandler} provides the {@link DataTypeManager} with a handler that is + * used to provide a disposition when a datatype conflict is detected during + * {@link DataTypeManager#resolve(DataType, DataTypeConflictHandler)} processing. + *
+ * Known Issue: resolve processing identifies a conflict on an outer datatype (e.g., Structure) + * before a resolve conflict decision has been made on its referenced datatypes. Depending + * upon the conflict handler used, this can result in duplicate conflict types once the full + * resolution is completed (see GP-3632). + */ public abstract class DataTypeConflictHandler { /** @@ -46,6 +56,7 @@ public abstract class DataTypeConflictHandler { return REPLACE_EMPTY_STRUCTS_OR_RENAME_AND_ADD_HANDLER; } }; + public abstract DataTypeConflictHandler getHandler(); } diff --git a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/StandAloneDataTypeManager.java b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/StandAloneDataTypeManager.java index 9bed6bbbd9..bb2d9eb94a 100644 --- a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/StandAloneDataTypeManager.java +++ b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/data/StandAloneDataTypeManager.java @@ -648,7 +648,7 @@ public class StandAloneDataTypeManager extends DataTypeManagerDB implements Clos "Program-architecture change not permitted"); } - if (!dbHandle.canUpdate()) { + if (readOnlyMode) { throw new ReadOnlyException("Read-only Archive: " + getName()); } diff --git a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/AttributeId.java b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/AttributeId.java index 281df9f5b1..a82880f468 100644 --- a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/AttributeId.java +++ b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/AttributeId.java @@ -235,6 +235,11 @@ public record AttributeId(String name, int id) { // public static final AttributeId ATTRIB_VARIANT = new AttributeId("variant", 143); // public static final AttributeId ATTRIB_VERSION = new AttributeId("version", 144); + // signature + public static final AttributeId ATTRIB_BADDATA = new AttributeId("baddata", 145); + public static final AttributeId ATTRIB_HASH = new AttributeId("hash", 146); + public static final AttributeId ATTRIB_UNIMPL = new AttributeId("unimpl", 147); + // public static final AttributeId ATTRIB_ADDRESS = new AttributeId("address", 148); public static final AttributeId ATTRIB_STORAGE = new AttributeId("storage", 149); diff --git a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/ElementId.java b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/ElementId.java index 51bfc12ef3..6575d10078 100644 --- a/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/ElementId.java +++ b/Ghidra/Framework/SoftwareModeling/src/main/java/ghidra/program/model/pcode/ElementId.java @@ -423,6 +423,20 @@ public record ElementId(String name, int id) { public static final ElementId ELEM_COMMAND_GETUSEROPNAME = new ElementId("command_getuseropname", COMMAND_GETUSEROPNAME); + // signature + public static final ElementId ELEM_BLOCKSIG = new ElementId("blocksig", 258); + public static final ElementId ELEM_CALL = new ElementId("call", 259); + public static final ElementId ELEM_GENSIG = new ElementId("gensig", 260); + public static final ElementId ELEM_MAJOR = new ElementId("major", 261); + public static final ElementId ELEM_MINOR = new ElementId("minor", 262); + public static final ElementId ELEM_COPYSIG = new ElementId("copysig", 263); + public static final ElementId ELEM_SETTINGS = new ElementId("settings", 264); + public static final ElementId ELEM_SIG = new ElementId("sig", 265); + public static final ElementId ELEM_SIGNATUREDESC = new ElementId("signaturedesc", 266); + public static final ElementId ELEM_SIGNATURES = new ElementId("signatures", 267); + public static final ElementId ELEM_SIGSETTINGS = new ElementId("sigsettings", 268); + public static final ElementId ELEM_VARSIG = new ElementId("varsig", 269); + public static final ElementId ELEM_SPLITDATATYPE = new ElementId("splitdatatype", 270); public static final ElementId ELEM_JUMPTABLEMAX = new ElementId("jumptablemax", 271); public static final ElementId ELEM_NANIGNORE = new ElementId("nanignore", 272); diff --git a/Ghidra/RuntimeScripts/Linux/support/bsim b/Ghidra/RuntimeScripts/Linux/support/bsim new file mode 100755 index 0000000000..6f97a9820c --- /dev/null +++ b/Ghidra/RuntimeScripts/Linux/support/bsim @@ -0,0 +1,25 @@ +#!/bin/bash +# +# Command-line script for interacting with a BSim database + +# maximum heap memory (may be increased) +MAXMEM=768M + +# launch mode (fg, bg, debug, debug-suspend) +LAUNCH_MODE=fg + +#set debug port for debug mode if used + +OS=`uname -s` + +SED=/bin/sed +if [ "$OS" = "Darwin" ]; then + SED=/usr/bin/sed +fi + +LAUNCH_DIR=`echo $0 | $SED -e 's/[^\/]*$//'` +if [ "$LAUNCH_DIR" = "" ]; then + LAUNCH_DIR="./"; +fi + +${LAUNCH_DIR}launch.sh $LAUNCH_MODE jdk "BSim" $MAXMEM "" ghidra.features.bsim.query.ingest.BSimLaunchable "$@" diff --git a/Ghidra/RuntimeScripts/Linux/support/bsim_ctl b/Ghidra/RuntimeScripts/Linux/support/bsim_ctl new file mode 100755 index 0000000000..0e9a2142b8 --- /dev/null +++ b/Ghidra/RuntimeScripts/Linux/support/bsim_ctl @@ -0,0 +1,30 @@ +#!/bin/bash +# +# Command-line script for controlling (start/stop) the BSim database + +# maximum heap memory (may be increased) +MAXMEM=768M + +# launch mode (fg, bg, debug, debug-suspend) +LAUNCH_MODE=fg + +#set debug port for debug mode if used + +OS=`uname -s` +MACHINE=`uname -m` + +SED=/bin/sed +if [ "$OS" = "Darwin" ]; then + SED=/usr/bin/sed +fi + +LAUNCH_DIR=`echo $0 | $SED -e 's/[^\/]*$//'` +if [ "$LAUNCH_DIR" = "" ]; then + LAUNCH_DIR="./"; +fi + +# Some JVM's with class data sharing enabled have issues with BSim starting with Ghidra's custom +# classloader, so we will disable sharing +VMARG_LIST="-Xshare:off" + +${LAUNCH_DIR}launch.sh $LAUNCH_MODE jdk "BSimControl" $MAXMEM "${VMARG_LIST}" ghidra.features.bsim.query.BSimControlLaunchable "$@" diff --git a/Ghidra/RuntimeScripts/Windows/support/bsim.bat b/Ghidra/RuntimeScripts/Windows/support/bsim.bat new file mode 100644 index 0000000000..fcc059ca19 --- /dev/null +++ b/Ghidra/RuntimeScripts/Windows/support/bsim.bat @@ -0,0 +1,23 @@ +:: Command-line script for interacting with a BSim database + +@echo off +setlocal + +:: Maximum heap memory may be changed if default is inadequate. This will generally be up to 1/4 of +:: the physical memory available to the OS. Uncomment MAXMEM setting if non-default value is needed. +::set MAXMEM=2G + +:: launch mode (fg, bg, debug, debug-suspend) +set LAUNCH_MODE=fg + +:: Sets LAUNCH_DIR to the directory that contains this file (bsim.bat). +:: LAUNCH_DIR will not contain a trailing slash. +:: +:: '% ~' dereferences the value in param 0 +:: 'd' - drive +:: 'p' - path (without filename) +:: '~0,-1' - removes trailing \ +set "LAUNCH_DIR=%~dp0" +set "LAUNCH_DIR=%LAUNCH_DIR:~0,-1%" + +call "%LAUNCH_DIR%\launch.bat" %LAUNCH_MODE% jdk BSim "%MAXMEM%" "" ghidra.features.bsim.query.ingest.BSimLaunchable %* diff --git a/Ghidra/RuntimeScripts/certification.manifest b/Ghidra/RuntimeScripts/certification.manifest index 52d9b254d8..69c8bd0a47 100644 --- a/Ghidra/RuntimeScripts/certification.manifest +++ b/Ghidra/RuntimeScripts/certification.manifest @@ -15,6 +15,8 @@ Linux/server/svrInstall||GHIDRA||||END| Linux/server/svrUninstall||GHIDRA||||END| Linux/support/GhidraGo/ghidraGo||GHIDRA||||END| Linux/support/analyzeHeadless||GHIDRA||||END| +Linux/support/bsim||GHIDRA||||END| +Linux/support/bsim_ctl||GHIDRA||||END| Linux/support/buildGhidraJar||GHIDRA||||END| Linux/support/buildNatives||GHIDRA||||END| Linux/support/convertStorage||GHIDRA||||END| @@ -30,6 +32,7 @@ Windows/server/svrUninstall.bat||GHIDRA||||END| Windows/support/GhidraGo/ghidraGo.bat||GHIDRA||||END| Windows/support/README_createPdbXmlFiles.txt||GHIDRA||||END| Windows/support/analyzeHeadless.bat||GHIDRA||||END| +Windows/support/bsim.bat||GHIDRA||||END| Windows/support/buildGhidraJar.bat||GHIDRA||||END| Windows/support/buildNatives.bat||GHIDRA||||END| Windows/support/convertStorage.bat||GHIDRA||||END| diff --git a/Ghidra/Test/IntegrationTest/src/test/java/ghidra/program/util/DataTypeCleanerTest.java b/Ghidra/Test/IntegrationTest/src/test/java/ghidra/program/util/DataTypeCleanerTest.java new file mode 100644 index 0000000000..51ffe1e866 --- /dev/null +++ b/Ghidra/Test/IntegrationTest/src/test/java/ghidra/program/util/DataTypeCleanerTest.java @@ -0,0 +1,218 @@ +/* ### + * IP: GHIDRA + * + * Licensed under the Apache License, Version 2.0 (the "License"); + * you may not use this file except in compliance with the License. + * You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ +package ghidra.program.util; + +import static org.junit.Assert.*; + +import org.junit.Before; +import org.junit.Test; + +import generic.test.AbstractGenericTest; +import ghidra.program.database.ProgramBuilder; +import ghidra.program.model.data.*; +import ghidra.program.model.listing.Program; + +public class DataTypeCleanerTest extends AbstractGenericTest { + + private Program program; + + private Structure structA; + private Structure structB; + private Structure structC; + + @Before + public void setUp() throws Exception { + ProgramBuilder builder = new ProgramBuilder("Test", ProgramBuilder._TOY); + program = builder.getProgram(); + + program.startTransaction("TEST"); + + /** + * /structA + * pack(disabled) + * Structure structA { + * 0 int 4 a0 "" + * 4 char[0] 0 az "" + * 5 byte 1 a1 "" + * 6 char[0] 0 aflex "" + * } + * Size = 6 Actual Alignment = 1 + */ + structA = new StructureDataType("structA", 0); + structA.add(IntegerDataType.dataType, "a0", null); + structA.add(new ArrayDataType(CharDataType.dataType, 0, -1), "az", null); + structA.add(DataType.DEFAULT); + structA.add(ByteDataType.dataType, "a1", null); + structA.add(new ArrayDataType(IntegerDataType.dataType, 0, -1), "aflex", null); + + /** + * /structB + * pack() + * Structure structB { + * 0 short 2 b0 "" + * 4 int:0(0) 0 "" + * 4 byte[0] 0 bfs "" + * 4 int:4(0) 1 bf1 "" + * 4 int:6(4) 2 bf2 "" + * 5 int:2(2) 1 bf3 "" + * 6 structA 6 b1 "" + * } + * Size = 12 Actual Alignment = 4 + */ + structB = new StructureDataType("structB", 0); + structB.setPackingEnabled(true); + structB.add(ShortDataType.dataType, "b0", null); + structB.addBitField(IntegerDataType.dataType, 0, null, null); // force integer alignment + structB.add(new ArrayDataType(ByteDataType.dataType, 0, -1), "bfs", null); + structB.addBitField(IntegerDataType.dataType, 4, "bf1", null); + structB.addBitField(IntegerDataType.dataType, 6, "bf2", null); + structB.addBitField(IntegerDataType.dataType, 2, "bf3", null); + structB.add(structA, "b1", null); + + /** + * /structC + * pack() + * Structure structC { + * 0 char 1 c0 "" + * 4 structB 12 c1 "" + * } + * Size = 16 Actual Alignment = 4 + */ + structC = new StructureDataType("structC", 0); + structC.setDescription("My structC"); + structC.setPackingEnabled(true); + structC.add(CharDataType.dataType, "c0", null); + structC.add(structB, "c1", null); + + } + + @Test + public void testClean() { + + DataTypeManager dtm = program.getDataTypeManager(); + DataOrganization dataOrganization = dtm.getDataOrganization(); + + try (DataTypeCleaner dtCleaner = new DataTypeCleaner(program.getDataTypeManager(), false)) { + + Structure cleanC = (Structure) dtCleaner.clean(structC); + assertEquals(structC.getCategoryPath(), cleanC.getCategoryPath()); + assertEquals(structC.getName(), cleanC.getName()); + assertEquals(structC.getDescription(), cleanC.getDescription()); + assertTrue(cleanC.isNotYetDefined()); + + Pointer ptr = new PointerDataType(structB); + Pointer cleanPtr = (Pointer) dtCleaner.clean(ptr); + assertEquals(dataOrganization.getPointerSize(), cleanPtr.getLength()); + DataType dt = cleanPtr.getDataType(); + assertTrue(dt instanceof Structure); + assertEquals(structB.getCategoryPath(), dt.getCategoryPath()); + assertEquals(structB.getName(), dt.getName()); + + } + } + + @Test + public void testCleanWithExisting1() { + + DataTypeManager dtm = program.getDataTypeManager(); + DataOrganization dataOrganization = dtm.getDataOrganization(); + + dtm.resolve(structC, null); + + try (DataTypeCleaner dtCleaner = new DataTypeCleaner(program.getDataTypeManager(), false)) { + + Structure cleanC = (Structure) dtCleaner.clean(structC); + assertEquals(structC.getCategoryPath(), cleanC.getCategoryPath()); + assertEquals(structC.getName(), cleanC.getName()); + assertEquals(structC.getDescription(), cleanC.getDescription()); + assertTrue(cleanC.isNotYetDefined()); + + Pointer ptr = new PointerDataType(structC); + Pointer cleanPtr = (Pointer) dtCleaner.clean(ptr); + assertEquals(dataOrganization.getPointerSize(), cleanPtr.getLength()); + DataType dt = cleanPtr.getDataType(); + assertTrue(dt instanceof Structure); + assertEquals(structC.getCategoryPath(), dt.getCategoryPath()); + assertEquals(structC.getName(), dt.getName()); + + } + } + + @Test + public void testCleanWithExisting2() { + + DataTypeManager dtm = program.getDataTypeManager(); + DataOrganization dataOrganization = dtm.getDataOrganization(); + + DataType resolvedDt = dtm.resolve(structC, null); + + try (DataTypeCleaner dtCleaner = new DataTypeCleaner(program.getDataTypeManager(), true)) { + + Structure cleanC = (Structure) dtCleaner.clean(structC); + assertEquals(structC.getCategoryPath(), cleanC.getCategoryPath()); + assertEquals(structC.getName(), cleanC.getName()); + assertEquals(structC.getDescription(), cleanC.getDescription()); + assertFalse(cleanC.isNotYetDefined()); + assertTrue(resolvedDt.isEquivalent(cleanC)); + + Pointer ptr = new PointerDataType(structC); + Pointer cleanPtr = (Pointer) dtCleaner.clean(ptr); + assertEquals(dataOrganization.getPointerSize(), cleanPtr.getLength()); + DataType dt = cleanPtr.getDataType(); + assertTrue(dt instanceof Structure); + assertEquals(structC.getCategoryPath(), dt.getCategoryPath()); + assertEquals(structC.getName(), dt.getName()); + assertFalse(dt.isNotYetDefined()); + assertTrue(resolvedDt.isEquivalent(dt)); + + } + } + + @Test + public void testCleanWithExisting3() { + + DataTypeManager dtm = program.getDataTypeManager(); + DataOrganization dataOrganization = dtm.getDataOrganization(); + + dtm.resolve(structA, null); + + try (DataTypeCleaner dtCleaner = new DataTypeCleaner(program.getDataTypeManager(), true)) { + + FunctionDefinition funcDef = + new FunctionDefinitionDataType(new CategoryPath("/Foo"), "Bar"); + Pointer ptr1 = new PointerDataType(structA); + Pointer ptr2 = new PointerDataType(structC); + funcDef.setArguments( + new ParameterDefinition[] { new ParameterDefinitionImpl("P1", ptr1, null) }); + funcDef.setReturnType(ptr2); + + FunctionDefinition cleanFuncDef = (FunctionDefinition) dtCleaner.clean(funcDef); + Pointer cleanPtr1 = (Pointer) cleanFuncDef.getArguments()[0].getDataType(); + assertTrue(ptr1.isEquivalent(cleanPtr1)); + assertEquals(dataOrganization.getPointerSize(), cleanPtr1.getLength()); + Pointer cleanPtr2 = (Pointer) cleanFuncDef.getReturnType(); + assertFalse(ptr2.isEquivalent(cleanPtr2)); + assertEquals(dataOrganization.getPointerSize(), cleanPtr2.getLength()); + DataType dt = cleanPtr2.getDataType(); + assertTrue(dt instanceof Structure); + assertEquals(structC.getCategoryPath(), dt.getCategoryPath()); + assertEquals(structC.getName(), dt.getName()); + assertTrue(dt.isNotYetDefined()); + + } + } + +} diff --git a/Ghidra/application.properties b/Ghidra/application.properties index 88274c64fd..2cef3a0a61 100644 --- a/Ghidra/application.properties +++ b/Ghidra/application.properties @@ -1,5 +1,5 @@ application.name=Ghidra -application.version=10.5 +application.version=11.0 application.release.name=DEV application.layout.version=1 application.gradle.min=7.3 diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_BSim_Command_Line.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_BSim_Command_Line.md new file mode 100644 index 0000000000..ca44856b92 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_BSim_Command_Line.md @@ -0,0 +1,73 @@ +# BSim Databases from the Command Line + +The ``bsim`` command-line utility, located in the ``support`` directory of a Ghidra distribution, is used to create, populate, and manage BSim databases. +It works for all BSim database backends. + +This utility offers a number of commands, many of which have several options. +In this section, we cover only a small subset of the possibilities. + +Note that running ``bsim`` with no arguments will print a detailed usage message. + +## Generating Signature Files + +The first step is to create signature files from the binaries in the Ghidra project. +Signature files are XML files which contain the BSim vectors and other metadata needed by the BSim server. + +**Important**: If you have the ``postgres_object_files`` project open in Ghidra, close it now. +Non-shared projects are locked when open, and the lock will prevent the signature-generating process from accessing the project. + +To generate the signature files, execute the following commands in a shell (adjust as necessary for Windows) + +```bash +cd /support +mkdir ~/bsim_sigs +./bsim generatesigs ghidra://postgres_object_files bsim=file://example ~/bsim_sigs +``` + +- The ``ghidra:/`` argument is the local project which holds the analyzed binaries. +Note that there is only one forward slash in the URL for a local project. +- The ``bsim=`` argument is the URL of the BSim database. +This command does not add any signatures to the database, but it does query the database for its settings. + +## Committing Signature Files + +Now, we commit the signatures to the BSim database with the following command (still in the ``support`` directory). + +```bash +./bsim commitsigs file://example ~/bsim_sigs +``` + +## Aside: Creating a Database + +We continue to use the database ``example``, so this step isn't necessary for the exercises. + +However, if we hadn't created ``example`` using a script, we could have used the following command: + +```bash +./bsim createdatabase file://example medium_nosize +``` +- ``medium_nosize`` is a database template. + - "medium" (vs. "large") affects the vector index and is not relevant to H2 databases. + - "nosize" means that size differences for varnodes of size four bytes and above are not incorporated into the BSim features. + This is necessary to allow matching between 32-bit and 64-bit code. +- The ``createdatabase`` command can also be used to create a BSim database on a PostgreSQL or Elasticsearch server, provided the servers are configured and running. +See the "BSim" entry in the Ghidra help for details. + +## Aside: Executable Categories and Function Tags + +It's worth a brief note about Executable Categories and Function Tags, although they are not used in any of the following exercises. + +A BSim database can record user-defined metadata about an executable or about functions within an executable. +Categories and tags can then be used as filter elements in a BSim query. +For example, you could restrict a BSim query to search only in executables of the category "OPEN_SOURCE" or to functions which have been tagged "COMPRESSION_FUNCTIONS". + +Executable categories in BSim are implemented using *program properties*, and function tags in BSim correspond to function tags in Ghidra. Properties and tags both have uses in Ghidra which are independent of BSim. +So, if we want a BSim database to record a particular category or tag, we must indicate that explicitly. + +For example, to inform the database that we wish to record the ``ORIGIN`` category, you would execute the command + +```bash +./bsim addexecategory file://example ORIGIN +``` + +Next Section: [Evaluating_Matches](BSimTutorial_Evaluating_Matches.md) diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Basic_Queries.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Basic_Queries.md new file mode 100644 index 0000000000..30556ad244 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Basic_Queries.md @@ -0,0 +1,160 @@ +# Basic BSim Queries + +In this section, we demonstrate some applications of our BSim database. + +## Registering a BSim Database + +In order to query the database, you must register it with Ghidra: + +1. From The Code Browser, Select **BSim -> Manage Servers**. +1. In the BSim Server Manager dialog, click the green plus. +1. Select the **File** radio button and use the chooser to select ``example.mv.db`` +1. Click **OK** +1. Click **Dismiss** to close the dialog. + +## How to Query a BSim Database + +Before presenting the exercises, we describe the general mechanics of querying a BSim database. + +### Initiating a BSim Query + +There are a number of ways to initiate a BSim query, including: + +- **BSim -> Search Functions...** from the Code Browser. +- Right-click in the Listing and select **BSim -> Search Functions...** +- Click on the BSim icon in the toolbar. + +For these cases, the function(s) being queried depend on the current selection. +If there is no selection, the function containing the current address is queried. +If there is a selection, all functions whose entry points are within the selection are queried. +For example, to query all functions in the program, first select all addresses in the program via ``Ctrl-A`` in the Listing window. + +It is also possible to initiate a BSim query from the Decompiler window. +Simply right-click on a function name token and select **BSim...** to query the corresponding function. +This action is available on the name token in the decompiled function's signature as well as tokens corresponding to names of callees. + +All of these actions bring up the *BSim Search Dialog*. + +### The BSim Search Dialog + +From the BSim Search Dialog, you can + +- Select which BSim database to query. +- Set query thresholds. +- Bound the number of results returned for each function. +- Set query filters. + +![](./images/bsim_search_dialog.png) + +#### Selecting a BSim Database + +To query a registered BSim database, select that server from the **BSim Server** drop-down. + +#### Setting Query Options + +**Similarity** and **confidence** are scores used to evaluate the relationship between two vectors. +The respective fields in the dialog set lower bounds for these values for the matches returned by BSim. + +- Similarity + - Formally, the similarity of a match is the cosine of the angle between the vectors. + - For BSim vectors, this value will always be between 0.0 and 1.0. + - The higher the similarity score, the closer the vectors. + +- Confidence + - Intuitively, confidence quantifies the meatiness of a match. + - Shared features increase this score and differing features decrease this score. + - Sharing rare features contributes more to this score than sharing common features. + - There is no upper bound for confidence when considered over all pairs of vectors. + However, if you fix a vector *v*, the greatest possible confidence score for a comparison involving *v* occurs when *v* is compared to itself. + The resulting confidence value is called the **self significance** of *v*. + +Confidence is used to judge the significance of a match. +For example, many executables contain a function which simply returns a constant value. +Given two executables, each with such a function, the similarity score between the corresponding BSim vectors will be 1.0. +However, the confidence score of the match will be quite low, indicating that it is not very significant that the two executables "share" this code. + +In general, setting the thresholds involves a tradeoff: lower values mean that the database is more likely to return legitimate matches with significant differences, but also more likely to return matches which simply happen to share some features by chance. +The results of a BSim query can be sorted by the similarity and/or confidence of each match, so a common practice is to set the thresholds relatively low and to examine the matches in descending sort order. + +The **Matches per Function** bound controls the number of results returned for a single function. +Note that in large collections, certain small or common functions might have substantial numbers of identical matches. + +#### Performing the Query + +Click the **Search** button in the dialog to perform a query. + +**Notes**: + +1. Filters are discussed in [BSim Filters](BSimTutorial_Filters.md). +1. After successfully issuing a query, you will also see a **Search Function(s)** action (without the ellipsis) in certain contexts. +This will perform a BSim query on the selected functions using the same parameters as the last query (skipping the BSim Seach Dialog). + +## Exercises: + +The database `example` contains vectors from a Linux executable used by Ghidra's GNU demangler. +Ghidra ships with several other versions of this executable. +We use these different versions to demonstrate some of the capabilities of BSim. + +**Note**: Use the default query settings and autoanalysis options for the exercises unless otherwise specified. + +### Exercise 1: Function Identification + +1. Import and analyze the binary ``/GPL/DemanglerGnu/os/win_x86_64/demangler_gnu_v2_41.exe``. + - This executable is based on the same source code as ``demangler_gnu_v2_41`` but compiled with Visual Studio instead of GCC. +1. Examine this binary in Ghidra and verify that the original function names are not present. + - Note that the function names **are** present in ``demangler_gnu_v2_41``. +1. Using the default query options, query `example` for matches to the function at ``140006760``. +1. You should see the following search results: + ![results](./images/basic_query.png) + - In this case, there is exactly one match, the similarity is 1.0, and the matching function has a non-default name (it won't always be this easy). + - **Note**: The results window has two tables: the function-level results (upper table) and the executable-level results (lower table). + The executable-level results are covered in [Executable-level Results](BSimTutorial_Exe_Results.md) +1. Right-click on the row of a match to see the available actions: + ![actions](./images/actions.png) +1. Select the **Compare Functions** action to bring up the side-by-side comparison. + - The **Listing View** tab shows the disassembly. + - The **Decompiler Diff View** tab shows the decompiled code. + - Differences in the code are automatically highlighted in blue. + - Either view can be toggled between a horizontal split and a vertical split using the drop-down menu. + - **Note**: We cover the Decompiler Diff View in greater detail in [Applying Function Signatures](BSimTutorial_Applying_Function_Signatures.md) +1. Examine the diff views to verify that the match is valid. +1. Using the `Apply Function Names and Namespaces` action, transfer the name from the search result to the queried function. + +TODO: explain why there are different apply actions + +### Exercise 2: Changes to the Source Code + +1. Import and analyze the executable ``/GPL/DemanglerGnu/os/linux_x86_64/demangler_gnu_v2_24``. + - This executable is based on an earlier version of the source code than the executable in ``example``. +1. Navigate to the function ``expandargv`` in ``demangler_gnu_v2_24`` and issue a BSim query. +1. What differences do you see in the decompiled code? +
In demangler_gnu_v2_41... Answer: The call to dupargv is now in an if clause (and decompiler creates a related local variable) and there are two additional calls to free.
+1. The relevant source files are included with the Ghidra distribution: + - ``/GPL/DemanglerGnu/src/demangler_gnu_v2_24/c/argv.c`` + - ``/GPL/DemanglerGnu/src/demangler/gnu_v2_41/c/argv.c`` +1. Verify that the differences you found are present in the source. + +### Exercise 3: Cross-architectural Matching + +1. Import and analyze the executable +``/GPL/DemanglerGnu/os/mac_arm_64/demangler_gnu_v2_41``. + - This executable is based on the same source code as the executable in `example` but compiled for a different architecture. + - **Note**: this file has the same name as the one used to populate the BSim database, so you will have to give the resulting Ghidra program a different name or import it into a different directory in your Ghidra project. +1. Navigate to ``_expandargv`` and issue a BSim query. What differences do you see regarding ``memmove`` and ``memcpy``? +
In the arm64 version... Answer: In the arm64_version, the compiler replaced these functions with __memmove_chk and __memcpy_chk. The __chk versions have an extra parameter related to preventing buffer overflows. Neither the names nor the bodies of callees are incorporated into BSim signatures, but the arguments of a call are, so this change partly explains why the BSim vectors are not identical.
+1. Examine the ``Listing View`` tab and verify that the architectures are different. + + +## A Remark on Query Thresholds and Indices + +Q: If you set the similarity and confidence thresholds to 0.0, will a BSim query return all of the functions in the database? + +A: No, because +- For indexed databases (i.e., PostgreSQL and Elasticsearch), the index is designed so that vector comparisons are only performed between vectors which are likely to be close. +Most vectors will not even be considered as potential matches for a given queried vector. +- Regardless of database backed, matches are only shown if the confidence score is above the confidence threshold of the query. +The interface will not allow you to set a negative confidence threshold, but confidence scores can be negative. +- The **Matches per Function** parameter also controls how many functions are returned. + +Next Section: [Ghidra from the Command Line](BSimTutorial_Ghidra_Command_Line.md) + diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Creating_Database_From_GUI.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Creating_Database_From_GUI.md new file mode 100644 index 0000000000..4241155068 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Creating_Database_From_GUI.md @@ -0,0 +1,28 @@ +# Creating and Populating a BSim Database from the Ghidra GUI + +This section explains how to create and populate an H2-backed BSim database from the Ghidra GUI. + +## Creating the Database + +To create a BSim database, first create a directory on your file system to contain the database. + +Next, perform the following steps from the Ghidra Code Browser: + +1. Run the Ghidra script ``CreateH2BSimDatabaseScript.java``. +1. In the resulting dialog: + 1. Enter "example" in the `Database Name` field. + 1. Select the new directory in the `Database Directory` field. + 1. Don't change any of the other fields. +1. Click OK. + +## Populating the Database + +We now populate the database with an executable which is contained in the Ghidra distribution. + +1. Import and analyze the executable ``/GPL/DemanglerGnu/os/linux_x86_64/demangler_gnu_v2_41`` using the default analysis options. +1. Run the Ghidra script ``AddProgramToH2BSimDatabaseScript.java`` on this program. + - The script will ask you to select an H2 database file. Use ``example.mv.db`` in the database directory. +1. In general you can run this script on other programs to add their signatures to this database, but that's not necessary for the exercises in the next section. + +Next Section: [Basic BSim Queries](BSimTutorial_Basic_Queries.md) + diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.md new file mode 100644 index 0000000000..44f3ede451 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Enabling.md @@ -0,0 +1,20 @@ +# Starting Ghidra and Enabling the BSim Plugin: + +To begin the tutorial, perform the following steps: + +1. Launch Ghidra. +1. Create a new non-shared project for this tutorial. +1. Launch the Code Browser. + +To enable BSim, perform the following steps: + +1. **File -> Configure** from the Code Browser. +1. Click on the ``Configure`` link of the ``BSim`` entry. +1. In the resulting dialog, ensure that the checkbox for ``BSimSearchPlugin`` is checked. + +![](./images/configure.png) + + Next Section: [Creating and Populating a BSim Database from the GUI](BSimTutorial_Creating_Database_From_GUI.md) + + + diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Evaluating_Matches.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Evaluating_Matches.md new file mode 100644 index 0000000000..486852a04b --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Evaluating_Matches.md @@ -0,0 +1,80 @@ +# Evaluating Matches and Transferring Information + +Summarizing what we've created over the last few sections, we now have: +1. A stripped executable (``postgres``). +1. A Ghidra project containing some object files *with debug information*[^1] used to build that executable. +1. A BSim database of containing the BSim signatures of the object files. + +[^1]: Having debug information isn't necessary to use BSim (as we've seen in a previous exercise), but it is convenient. + +We now demonstrate using BSim to help reverse engineer ``postgres``. +While doing this, we'll showcase some of the features available in the decompiler diff view. + +## Exercise: Exploring the Highlights + +Import and analyze the stripped `postgres` executable into the tutorial project, then perform the following steps: + +1. Select all functions in `postgres` via Ctrl-A in the Listing. +1. Perform a BSim query of the database ``example``. + - **Note:** We use the results of this query in the following few exercises. + If don't close the BSim search results window, you won't have to issue the query again. +1. Sort the rows by confidence and find the row with ``grouping_planner`` as the matching function. +The corresponding function in `postgres` should have a default name. +1. Examine this match in the side-by-side decompiler view. +Note that the matching function has better data type information due to the debug information. +1. Q: Why does the placement of the `double` argument between the functions? +
Answer Floating point values and integer/pointer values are passed in separate sets registers. + Neither ordering is wrong since both are consistent with the instructions of the function. + The debug info records a specific signature (and ordering) for the function, which Ghidra applies. + In the version without debug information, the decompiler used heuristics to determine the function's signature.
+ +For matches with a fair number of differences, the decompiler diff panel can get pretty colorful. +Furthermore, as you click around, tokens will gain and lose highlight of various colors. +It's worth giving a brief explanation of when highlighting happens and what the different colors mean. +Some terminology: if you click on a token in a decompiler panel, that token becomes the *focused token*. + +The colors: + +- Blue is used to highlight differences between the two functions. +- Pink is used to highlight the focused token and its match. +- Lavender is used to highlight the focused token when it does not have a match. +- Orange is used to highlight the focused token when it is ineligible for match. +Certain tokens, such as whitespace tokens or tokens used in variable declarations, are never assigned matching tokens. + +## Exercise: Locking and Unlocking Scrolling + + +Before moving on, experiment with locking and unlocking scrolling. + +## Exercise: Comparing Callees + +The token matching algorithm matches a function call in one program to a function call in another by considering the data flow into and out of the ``CALL`` instruction, but it does not do anything with the bodies of the callees. +However, given a matched pair of calls, you can bring up a new comparison window and compare their bodies manually. + +Ctrl f in left view +FUN_ +find something + + + + + +## Exercise: Transferring Signatures + +1. Transfer the signatures to the queried function via either: + - The `Apply Function Signature to Other Side` action in the diff window. + - The `Apply Function Names, Namespaces, and Signatures` action in the BSim Search Results window. + +**Warning**: You should be absolutely certain that the datatypes are the same before applying signatures. +If there have been any changes to a datatype's definition, you could end up bringing incorrect datatypes into a program, even using BSim matches with 1.0 similarity. + +# Exercise: Multiple Comparisons + + + + + +In the next section, we discuss the Executable Results table. + + +Next Section: [Executable-level Results](BSimTutorial_Exe_Results.md) \ No newline at end of file diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Exe_Results.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Exe_Results.md new file mode 100644 index 0000000000..81e8543d9b --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Exe_Results.md @@ -0,0 +1,37 @@ +# From Matching Functions to Matching Executables + +In this section, we discuss the Executable results table. + +Using the results window from the previous query, sort the Executable results table +by "Function Count" (i.e., the number of results which are in a given executable). You should see the following: + +![](./images/exe_results.png) + +If you select a single row in the table and right-click on it, you will see the following actions: + +![](./images/exe_results_actions.png) + +- **Load Executable** will open a read-only copy of the program in the Code Browser. +- **Filter on this Executable** applies a filter which restricts the matches shown in the Function Matches table to matches which occur in the given executable. + +## Exercise + +1. If you haven't already, sort the Executable results by descending Function Count. +What position is `demangler_gnu_v2_33_1`? + -
A: 7
+1. The Confidence column shows the sum of the confidence scores of all matches into each executable. Sort the Executable results by descending confidence and observe that `demangler_gnu_v2_33_1` is now much further down the list. + -
What could explain this? + If there are many function matches but the sum of all the confidences is relatively low, + it is likely that many of the matches involve small functions. For such a match, it is + more likely that the functions agree by chance rather than being derived from the same + source code. +
+1. In the Executable match table, right click on `demangler_gnu_v2_33_1` and apply the filter +action. Sort the filtered function matches by descending confidence. Starting at the top, +perform some code comparisons and convince yourself that the given explanation is correct. + - Note: You can remove the filter using the "Settings" icon in the upper right. We'll discuss this further in [BSim Filters](./BSimTutorial_Filters.md) + +In the next section, describe a technique to restrict queries to functions which are likely to +have "interesting" matches. + +Next Section: [Overview Queries](BSimTutorial_Overview.md) \ No newline at end of file diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Filters.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Filters.md new file mode 100644 index 0000000000..adf428fdb4 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Filters.md @@ -0,0 +1,36 @@ +# BSim Filters + +There are a number of filters that can be applied to BSim queries, involving names, architectures, +compilers, ingest dates, and many other attributes. + +Filter be can applied *server-side* or *client-side*. Server-side filters affect the results sent +to Ghidra from a BSim server. Client-side filters apply to the BSim Search results table and can +be added and removed at will. However, to "undo" a server-side filter, you have to issue an +additional BSim query without the filter. + +Note that overview queries cannot be filtered. + +Server-side filters can be applied using the `Filters` drop-down in the BSim Search dialog. + +## Exercise: Filters + +1. Select all functions in `postgres` and bring up the BSim Search dialog. +1. Use the default query bounds. +1. Apply an `Executable name does not equal` filter with `demangler_gnu_v2_33_1` as the name to +exclude. +1. Perform the query and verify that `demangler_gnu_v2_33_1` is not in the list of executables +with matches. +

+ +

+1. Using the `Search Info` icon, you can see what server-side filters were applied to the query. +Verify that this information is correct. +

+ +

+1. Using the `Filter Results` icon, you can apply client-side filters to the query results. +Experiment with applying and removing some client-side filters. + + +Next Section: [Scripting and Visualization](BSimTutorial_Scripting.md) + diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.md new file mode 100644 index 0000000000..32fcbeb1bb --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.md @@ -0,0 +1,52 @@ +# Ghidra Analysis from the Command Line + +For the remaining exercises, we need to populate our BSim database with a number of binaries. +We'd like a consistent set of binaries for the tutorial, but we don't want to clutter the Ghidra distribution with dozens of additional executables that aren't actually used by the codebase. +Fortunately, the BSim plugin includes a script for building the PostgreSQL backend, and that build process creates hundreds of object files. +So we can just build PostgreSQL and harvest the object files we need. + +**Note**: For the tutorial, we continue to use the H2 BSim backend. +We do not run any PostgreSQL code, we simply analyze some files produced when building PostgreSQL. + +Note that these files must be built on a machine running Linux. +Windows users can build these files in a Linux virtual machine. + +To build the files, execute the following commands in a shell: [^1] + +[^1]: You may need to install additional packages and/or change some build options in order for PostgreSQL to build successfully. +The error messages are generally informative. See the comments in ``make-postgres.sh``. + +```bash +cd /Features/BSim +export CFLAGS="-O2 -g" +./make-postgres.sh +mkdir ~/postgres_object_files +cd build +find . -name pl*.o -exec cp {} ~/postgres_object_files/ \; +cd os/linux_x86_64/postgresql/bin +strip -s postgres +``` + +To continue on Windows, transfer the ``~/postgres_object_files`` directory and the (stripped) ``postgres`` executable to your Windows machine. + + +## Importing and Analyzing the Exercise Files + +Now that we have the executables, we can analyze them with the headless analyzer[^2]. +The headless analyzer is distinct from BSim, but using it is the only feasible way to analyze substantial numbers of binaries. + +[^2]: The headless analyzer has its own documentation: ``/support/analyzeHeadlessREADME.html``. + +To analyze the files in Linux, execute the following commands in a shell. + +```bash +cd /support +./analyzeHeadless postgres_object_files -import ~/postgres_object_files/* +``` +(On windows, use ``analyzeHeadless.bat`` and adjust paths accordingly.) + +This will create a local Ghidra project called ``postgres_object_files`` in the directory ````. + + +Next Section: [BSim from the Command Line](BSimTutorial_BSim_Command_Line.md) + diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Intro.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Intro.md new file mode 100644 index 0000000000..2405e78411 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Intro.md @@ -0,0 +1,91 @@ +# Introduction to BSim + +As you've reverse engineered software, you've likely asked the following questions: + +- Which libraries were statically linked into this executable? +- Does this executable share some code with another executable that I've analyzed? +- What are the differences between version 1 and version 2 of a given executable? +- Does this executable share code with another executable in a large collection of binaries? +- Was this function pulled from an open-source library? + +BSim is intended to help with these questions (and others) by providing a way to search collections of binaries for similar, but not necessarily identical, functions. + +# How Does BSim Work? + +The idea behind BSim is to generate a *feature vector* for each function in a binary. +The vectors are generated by Ghidra's decompiler. +Each feature represents a small piece of data flow and/or control flow of the associated function. +The decompiler normalizes the feature vector representation so that different, but functionally equivalent, pieces of code often produce the same features. +Certain attributes, such as values of constants, names of registers, and data types, are intentionally not incorporated into the features. + +BSim vectors are compared using *cosine similarity*. +Discrepancies between the vectors for ``foo`` and ``bar`` which are caused by differences in compilers, target architectures, and/or small changes to the source code typically result in vectors which are close but not identical. + +BSim vectors can be stored in a dedicated database. +BSim databases intended to hold large[^1] numbers of vectors maintain an index based on *locality-sensitive hashing*. +The index drastically reduces the number of vector comparisons needed and allows for rapid retrieval of results. + +[^1]: Creating a database requires a *database template*, which determines the specifics of the index. Currently, Ghidra provides a *medium* template, intended for +databases holding up to 10 million unique vectors, and a *large* template, intended for databases holding up to 100 million unique vectors. + +Querying ``foo`` against a BSim database typically yields a number of potential matches. +Each individual match for ``foo`` can be compared to `foo` in a side-by-side view, and certain information (such as function name) can be quickly transferred from a match to ``foo``. + +We frequently call BSim vectors the *BSim signature* of a function, or just the *signature* when the context is clear. + +# Why "BSim"? + +We can think of each feature as representing a small piece of the *behavior* of a function, analogous to a snippet of source code. +Functions whose BSim vectors are close typically have many features in common, that is, they have *similar behavior*. +Hence the name "BSim": **B**ehavioral **Sim**iliarity. + +# BSim Clients, BSim Databases, and Ghidra Projects + +Using BSim involves the following components: + +- A *BSim Client*, i.e., an instance of Ghidra with the BSim plugin enabled. + - This is where the reverse engineering happens. +- A *BSim Database*, which stores the BSim signatures. + - Also stores some metadata about each function and the containing executable. + - In particular, stores the ghidra:// URL of the associated Ghidra program. + - Does not store disassembly or decompiled functions. +- A *Ghidra Project*, which stores the analyzed programs used to populate the BSim database. + - Given a BSim match, the BSim client can use the ghidra:// URL to retrieve a program from a Ghidra project for side-by-side comparisons. + - Note that a single BSim database can reference multiple Ghidra projects. + +# Database Backends + +There are three supported database backends for BSim: + +1. PostgreSQL + + - The Ghidra distribution includes the source for a PostgreSQL plugin for BSim along with a + build script. + - Users must download the PostgreSQL source. + - Populated from shared Ghidra projects (i.e., requires a Ghidra server). + - Server not supported on Windows (no restriction on clients). + +2. Elasticsearch + + - The ``BSimElasticPlugin`` extension contains an Elasticsearch plugin for BSim. + - This plugin must be installed into an existing Elasticsearch database. + - Populated from shared Ghidra projects. + +3. H2 + + - Simplest way to use BSim: + - Backed by files on the user's machine (don't need to install database server). + - Can be created and populated quickly. + - Supported on all platforms. + - Does not support large collections of binaries or multiple users. + - Can be populated from non-shared (local) or shared Ghidra projects. + + Next Section: [Starting Ghidra and Enabling BSim](BSimTutorial_Enabling.md) + + + + + + + + diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Overview.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Overview.md new file mode 100644 index 0000000000..1a24895b1b --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Overview.md @@ -0,0 +1,57 @@ +# Overview Queries + +An **Overview Query** queries a BSim database for the number of matches to each +function in an executable. The matching functions themselves are not returned. +Similarity and Confidence thresholds apply to an Overview query, but the +"Matches per Function" bound does not. + +To perform an Overview Query, select `BSim -> Perform Overview...` from the Code +Browser. + +## Exercise 1: Hit Counts and Self-Similarities + +1. Perform an Overview query on `postgres` using the default query bounds. You should see +the following result: +![](./images/overview_window.png) +1. Sort the table by the "Hit Count" column in ascending order. Typically, the functions with the largest hit counts will have low self-similarity. Verify that that is the case for this table. +1. Q: Examine the functions with the highest hit count. Why are there so many matches, and +why do they all have the same BSim feature vector? + -
A: These functions simply return constants. BSim feature vectors +incorporate the fact that varnode is constant but do not incorporate the specific value.
+ +## Exercise 2: Selections and Queries + +Using the hit count column, it is possible to exclude functions with large numbers of matches. + +1. In the Overview Table, select all functions whose hit count is 5 or less. +1. Right-click on the selection and perform the `Search Selected Functions` action. Sort the +query results by `Function Count` and verify that `demangler_gnu_v2_33_1` is far down the list. + +## Exercise 3: Vector Hashes + +Suppose `foo` and `bar` have the same number of hits in the Overview table. There are two +possibilities: +- `foo` and `bar` have distinct feature vectors which happen to have the same number of matches. +- `foo` and `bar` have the same feature vector. + +An optional column, `Vector Hash`, can be used to distinguish between these two cases. + +1. Enable the `Vector Hash` Column in the Overview Table. +1. Sort the hit count column in ascending order, (multi)sort the Self Significance column in +descending order, then (multi)sort the Vector Hash column in ascending order. +1. Q: What are the first functions in the table with the same vector hash? + -
A: `ts_headline_json_byid_opt` and `ts_headline_jsob_byid_opt` +
+1. Examine the decompiled code of these two functions and verify that they should have identical +BSim vectors. + + + + + + + + + + +Next Section: [Queries and Filters](BSimTutorial_Filters.md) \ No newline at end of file diff --git a/GhidraDocs/GhidraClass/BSim/BSimTutorial_Scripting.md b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Scripting.md new file mode 100644 index 0000000000..1423fcded4 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/BSimTutorial_Scripting.md @@ -0,0 +1,23 @@ +# Scripting and Visualization + +Finally, we briefly mention a few other topics related to BSim. + +## Scripting BSim + +There are are number of example scripts in the ``BSim`` script category, which demonstrate how to interact with BSim programmatically: + +![](./images/script_manager.png) + +## Visualizing Features + +Finally, if you'd like to see the particular BSim features in a function, you can use the BSim Feature Visualizer. +This plugin allows you to highlight regions of the decompiled code corresponding to a particular feature and to display a graph representing the feature. + +To use this plugin, first enable the ``BSimFeatureVisualizerPlugin`` via **File -> Configure ** from the Code Browser. +You can then bring it via **BSim -> BSim Feature Visualizer**. + +![](./images/feature_visualizer.png) + +This is the end of the tutorial. + +[Return to the Beginning](README.md) \ No newline at end of file diff --git a/GhidraDocs/GhidraClass/BSim/README.md b/GhidraDocs/GhidraClass/BSim/README.md new file mode 100644 index 0000000000..168fdf7ec4 --- /dev/null +++ b/GhidraDocs/GhidraClass/BSim/README.md @@ -0,0 +1,22 @@ +# BSim Tutorial + +BSim is a Ghidra plugin for finding structurally similar functions in (potentially large) collections of binaries. +It is based on Ghidra's decompiler and can find matches across compilers, architectures, and/or small changes to source code. + +This tutorial demonstrates how create a small BSim database and walks through some typical use cases. + +**Detailed information about BSim can be found in the "BSim" entry of the Ghidra Help**. + +1. [Introduction to BSim](BSimTutorial_Intro.md) +1. [Enabling BSim](BSimTutorial_Enabling.md) +1. [Creating and Populating a BSim Database from the GUI](BSimTutorial_Creating_Database_From_GUI.md) +1. [Basic BSim Queries](BSimTutorial_Basic_Queries.md) +1. [Ghidra from the Command Line](BSimTutorial_Ghidra_Command_Line.md) +1. [BSim from the Command Line](BSimTutorial_BSim_Command_Line.md) +1. [Evaluating Matches](BSimTutorial_Evaluating_Matches.md) +1. [From Matching Functions to Matching Executables](BSimTutorial_Exe_Results.md) +1. [Overview Queries](BSimTutorial_Overview.md) +1. [BSim Filters](BSimTutorial_Filters.md) +1. [Scripting and Visualization](BSimTutorial_Scripting.md) + +Next Section: [Introduction to BSim](BSimTutorial_Intro.md) diff --git a/GhidraDocs/GhidraClass/BSim/images/actions.png b/GhidraDocs/GhidraClass/BSim/images/actions.png new file mode 100644 index 0000000000..919e8410a0 Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/actions.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/basic_query.png b/GhidraDocs/GhidraClass/BSim/images/basic_query.png new file mode 100644 index 0000000000..3c2b154c05 Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/basic_query.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/bsim_search_dialog.png b/GhidraDocs/GhidraClass/BSim/images/bsim_search_dialog.png new file mode 100644 index 0000000000..330b189b04 Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/bsim_search_dialog.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/configure.png b/GhidraDocs/GhidraClass/BSim/images/configure.png new file mode 100644 index 0000000000..f27751117c Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/configure.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/exe_results.png b/GhidraDocs/GhidraClass/BSim/images/exe_results.png new file mode 100644 index 0000000000..119d6653ed Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/exe_results.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/exe_results_actions.png b/GhidraDocs/GhidraClass/BSim/images/exe_results_actions.png new file mode 100644 index 0000000000..2787644d57 Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/exe_results_actions.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/feature_visualizer.png b/GhidraDocs/GhidraClass/BSim/images/feature_visualizer.png new file mode 100644 index 0000000000..c5b73322c6 Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/feature_visualizer.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/filter_results.png b/GhidraDocs/GhidraClass/BSim/images/filter_results.png new file mode 100644 index 0000000000..71b903a739 Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/filter_results.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/overview_window.png b/GhidraDocs/GhidraClass/BSim/images/overview_window.png new file mode 100644 index 0000000000..c9b1073474 Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/overview_window.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/script_manager.png b/GhidraDocs/GhidraClass/BSim/images/script_manager.png new file mode 100644 index 0000000000..295704e53b Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/script_manager.png differ diff --git a/GhidraDocs/GhidraClass/BSim/images/search_info.png b/GhidraDocs/GhidraClass/BSim/images/search_info.png new file mode 100644 index 0000000000..a27944762a Binary files /dev/null and b/GhidraDocs/GhidraClass/BSim/images/search_info.png differ diff --git a/GhidraDocs/certification.manifest b/GhidraDocs/certification.manifest index cf70968809..6636b71d05 100644 --- a/GhidraDocs/certification.manifest +++ b/GhidraDocs/certification.manifest @@ -19,6 +19,29 @@ GhidraClass/AdvancedDevelopment/GhidraAdvancedDevelopment.html||GHIDRA|||This fi GhidraClass/AdvancedDevelopment/GhidraAdvancedDevelopment_withNotes.html||Public Domain|||Slight modification of code that is available for distribution, without restrictions, (original extremely permissive wtf license allows us to change IP to Public Domain),from https://github.com/paulrouget/dzslides.|END| GhidraClass/AdvancedDevelopment/Images/GhidraLogo64.png||GHIDRA||||END| GhidraClass/AdvancedDevelopment/Images/highLevelClasses.png||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_BSim_Command_Line.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Basic_Queries.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Creating_Database_From_GUI.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Enabling.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Evaluating_Matches.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Exe_Results.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Filters.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Ghidra_Command_Line.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Intro.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Overview.md||GHIDRA||||END| +GhidraClass/BSim/BSimTutorial_Scripting.md||GHIDRA||||END| +GhidraClass/BSim/README.md||GHIDRA||||END| +GhidraClass/BSim/images/actions.png||GHIDRA||||END| +GhidraClass/BSim/images/basic_query.png||GHIDRA||||END| +GhidraClass/BSim/images/bsim_search_dialog.png||GHIDRA||||END| +GhidraClass/BSim/images/configure.png||GHIDRA||||END| +GhidraClass/BSim/images/exe_results.png||GHIDRA||||END| +GhidraClass/BSim/images/exe_results_actions.png||GHIDRA||||END| +GhidraClass/BSim/images/feature_visualizer.png||GHIDRA||||END| +GhidraClass/BSim/images/filter_results.png||GHIDRA||||END| +GhidraClass/BSim/images/overview_window.png||GHIDRA||||END| +GhidraClass/BSim/images/script_manager.png||GHIDRA||||END| +GhidraClass/BSim/images/search_info.png||GHIDRA||||END| GhidraClass/Beginner/Images/GhidraLogo64.png||GHIDRA||||END| GhidraClass/Beginner/Introduction_to_Ghidra_Student_Guide.html||GHIDRA|||This file contains mostly Ghidra content, but also includes code that is available for distribution, without restrictions, from https://github.com/paulrouget/dzslides.|END| GhidraClass/Beginner/Introduction_to_Ghidra_Student_Guide_withNotes.html||Public Domain|||Slight modification of code that is available for distribution, without restrictions, (original extremely permissive wtf license allows us to change IP to Public Domain),from https://github.com/paulrouget/dzslides.|END| diff --git a/gradle/support/fetchDependencies.gradle b/gradle/support/fetchDependencies.gradle index 6def2bf03e..34ca2d2be5 100644 --- a/gradle/support/fetchDependencies.gradle +++ b/gradle/support/fetchDependencies.gradle @@ -87,6 +87,12 @@ ext.deps = [ sha256: "4dae732a535846ae5dfab753e82a4d5f93ad9a05a065e2172bb9774a1b15453a", destination: file("${DEPS_DIR}/GhidraServer") ], + [ + name: "postgresql-15.3.tar.gz", + url: "https://ftp.postgresql.org/pub/source/v15.3/postgresql-15.3.tar.gz", + sha256: "086d38533e28747966a4d5f1e78ea432e33a78f21dcb9133010ecb5189fad98c", + destination: file("${DEPS_DIR}/BSim") + ], [ name: "PyDev 6.3.1.zip", url: "https://sourceforge.net/projects/pydev/files/pydev/PyDev%206.3.1/PyDev%206.3.1.zip", diff --git a/licenses/PostgresqlJDBC_License.txt b/licenses/PostgresqlJDBC_License.txt new file mode 100644 index 0000000000..5d3a0018c7 --- /dev/null +++ b/licenses/PostgresqlJDBC_License.txt @@ -0,0 +1,23 @@ +Copyright (c) 1997, PostgreSQL Global Development Group +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: + +1. Redistributions of source code must retain the above copyright notice, + this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright notice, + this list of conditions and the following disclaimer in the documentation + and/or other materials provided with the distribution. + +THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" +AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE +IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE +ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE +LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR +CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF +SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS +INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN +CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) +ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. diff --git a/licenses/Postgresql_License.txt b/licenses/Postgresql_License.txt new file mode 100644 index 0000000000..1681966945 --- /dev/null +++ b/licenses/Postgresql_License.txt @@ -0,0 +1,24 @@ +PostgreSQL Database Management System +(formerly known as Postgres, then as Postgres95) + +Portions Copyright (c) 1996-2023, PostgreSQL Global Development Group + +Portions Copyright (c) 1994, The Regents of the University of California + +Permission to use, copy, modify, and distribute this software and its +documentation for any purpose, without fee, and without a written agreement +is hereby granted, provided that the above copyright notice and this +paragraph and the following two paragraphs appear in all copies. + +IN NO EVENT SHALL THE UNIVERSITY OF CALIFORNIA BE LIABLE TO ANY PARTY FOR +DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING +LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS +DOCUMENTATION, EVEN IF THE UNIVERSITY OF CALIFORNIA HAS BEEN ADVISED OF THE +POSSIBILITY OF SUCH DAMAGE. + +THE UNIVERSITY OF CALIFORNIA SPECIFICALLY DISCLAIMS ANY WARRANTIES, +INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY +AND FITNESS FOR A PARTICULAR PURPOSE. THE SOFTWARE PROVIDED HEREUNDER IS +ON AN "AS IS" BASIS, AND THE UNIVERSITY OF CALIFORNIA HAS NO OBLIGATIONS TO +PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS. + diff --git a/licenses/certification.manifest b/licenses/certification.manifest index 66f4e56893..2740ad0cb8 100644 --- a/licenses/certification.manifest +++ b/licenses/certification.manifest @@ -26,5 +26,7 @@ MIT.txt||LICENSE||||END| Modified_Nuvola_Icons_-_LGPL_2.1.txt||LICENSE||||END| Nuvola_Icons_-_LGPL_2.1.txt||LICENSE||||END| Oxygen_Icons_-_LGPL_3.0.txt||LICENSE||||END| +PostgresqlJDBC_License.txt||LICENSE||||END| +Postgresql_License.txt||LICENSE||||END| Public_Domain.txt||LICENSE||||END| Tango_Icons_-_Public_Domain.txt||LICENSE||||END|