搜索结果：AST-based

暂无摘要（点击查看原文获取完整内容）

Programming languages function as a medium for expressing instructions in the context of computer program creation. A prevalent convention in programming is categorizing source code into fragments, a strategy that enhances understanding and simplifies the upkeep process. This research study presents a comparative examination of two distinct models utilized to classify source code. The first model employs a neural network approach that heavily relies on Syntax Trees (ASTs). In contrast, the second model is built on a neural network framework that utilizes the tokenization process. The AST-based model employs a methodology that entails decomposing extensive ASTs into smaller clusters of statement trees. These clusters are subsequently transformed into vectors by capturing the lexical and syntactical characteristics of the statements. The naturalness of certain statements is captured by employing a bidirectional Recurrent Neural Network (RNN) model, which utilizes a collection of statement vectors. In contrast, the token-based paradigm diverges from ASTs and emphasizes tokens obtained through tokenization. The tokens undergo preprocessing and vectorization before inputting into a bidirectional RNN model to evaluate the naturalness of the statements. To choose the optimal model, the accuracy of the two models are compared.

We focus on Text-to-SQL semantic parsing from the perspective of retrieval-augmented generation.Motivated by challenges related to the size of commercial database schemata and the deployability of business intelligence solutions, we propose ASTRES that dynamically retrieves input database information and uses abstract syntax trees to select few-shot examples for in-context learning.Furthermore, we investigate the extent to which an in-parallel semantic parser can be leveraged for generating approximated versions of the expected SQL queries, to support our retrieval.We take this approach to the extreme-we adapt a model consisting of less than 500M parameters, to act as an extremely efficient approximator, enhancing it with the ability to process schemata in a parallelised manner.We apply ASTRES to monolingual and crosslingual benchmarks for semantic parsing, showing improvements over state-of-the-art baselines.Comprehensive experiments highlight the contribution of modules involved in this retrieval-augmented generation setting, revealing interesting directions for future work.

We introduce ACER, an AST-based call graph generator framework. ACER leverages tree-sitter to interface with any language. We opted to focus on generators that operate on abstract syntax trees (ASTs) due to their speed and simplicitly in certain scenarios; however, a fully quantified intermediate representation usually provides far better information at the cost of requiring compilation. To evaluate our framework, we created two context-insensitive Java generators and compared them to existing open-source Java generators.Code: https://github.com/WM-SEMERU/ACER

Design patterns (DPs) provide reusable and general solutions for frequently encountered problems. Patterns are important to maintain the structure and quality of software products, in particular in large and distributed systems like automotive software. Modern language models (like Code2Vec or Word2Vec) indicate a deep understanding of programs, which has been shown to help in such tasks as program repair or program comprehension, and therefore show promise for DPR in industrial contexts. The models are trained in a self-supervised manner, using a large unlabelled code base, which allows them to quantify such abstract concepts as programming styles, coding guidelines, and, to some extent, the semantics of programs. This study demonstrates how two language models—Code2Vec and Word2Vec, trained on two public automotive repositories, can show the separation of programs containing specific DPs. The results show that the Code2Vec and Word2Vec produce average F1-scores of 0.781 and 0.690 on open-source Java programs, showing promise for DPR in practice.

暂无摘要（点击查看原文获取完整内容）

In recent years, PowerShell has been widely used in cyber attacks and malicious PowerShell scripts can easily evade the detection of anti-virus software through obfuscation. Existing deobfuscation tools often fail to recover obfuscated scripts correctly due to imprecise obfuscation identification, improper recovery and wrong replacement. In this paper, we propose an AST-based and semantics-preserving deobfuscation approach, Invoke-Deobfuscation. It utilizes recoverable nodes of Abstract Syntax Tree to identify obfuscated pieces precisely, simulates the recovery process through Invoke function and variable tracing, and replaces obfuscated pieces in place to keep the original semantics. We build a large evaluation dataset containing 39,713 wild PowerShell scripts. Compared with the state-of-the-art tools, the experimental results show Invoke-Deobfuscation performs most efficiently. It recovers much more key information than others and significantly reduces samples’ obfuscation score, on average, by 46%. Moreover, 100% of Invoke-Deobfuscation’s results have the same network behavior as the original scripts.

Optimizing compilers targeting modern multi-core machines require complex program restructuring to expose the best combinations of coarse- and fine-grain parallelism and data locality. The polyhedral compilation model has provided significant advancements in the seamless handling of compositions of loop transformations, thereby exposing multiple levels of parallelism and improving data reuse. However, it usually implements abstract optimization objectives, for example "maximize data reuse", which often does not deliver best performance, e.g., The complex loop structures generated can be detrimental to short-vector SIMD performance. In addition, several key transformations such as pipeline-parallelism and unroll-and-jam are difficult to express in the polyhedral framework. In this paper, we propose a novel optimization flow that combines polyhedral and syntactic/AST-based transformations. It generates high-performance code that contains regular loops which can be effectively vectorized, while still implementing sufficient parallelism and data reuse. It combines several transformation stages using both polyhedral and AST-based transformations, delivering performance improvements of up to 3× over the PoCC polyhedral compiler on Intel Nehalem and IBM Power7 multicore processors.

Code clone detection aims to find functionally similar code fragments, which is becoming more and more important in the field of software engineering. Many code clone detection methods have been proposed, among which tree-based methods are able to handle semantic code clones. However, these methods are difficult to scale to big code due to the complexity of tree structures. In this paper, we design Amain, a scalable tree-based semantic code clone detector by building Markov chains models. Specifically, we propose a novel method to transform the original complex tree into simple Markov chains and measure the distance of all states in these chains. After obtaining all distance values, we feed them into a machine learning classifier to train a code clone detector. To examine the effectiveness of Amain, we evaluate it on two widely used datasets namely Google Code Jam and BigCloneBench. Experimental results show that Amain is superior to nine state-of-the-art code clone detection tools (i.e., SourcererCC, RtvNN, Deckard, ASTNN, TBCNN, CDLH, FCCA, DeepSim, and SCDetector).

暂无摘要（点击查看原文获取完整内容）

Research on API migration and language conversion can be informed by empirical data about API usage. For instance, such data may help with designing and defending mapping rules for API migration in terms of relevance and applicability. We describe an approach to large-scale API-usage analysis of open-source Java projects, which we also instantiate for the Source-Forge open-source repository in a certain way. Our approach covers checkout, building, tagging with metadata, fact extraction, analysis, and synthesis with a large degree of automation. Fact extraction relies on resolved (type-checked) ASTs. We describe a few examples of API-usage analysis; they are motivated by API migration. These examples are concerned with analysing API footprint (such as the numbers of distinct APIs used in a project), API coverage (such as the percentage of methods of an API used in a corpus), and framework-like vs. class-library-like usage.

Most programmers rely on visual tools (block-based editors, auto-indentation, bracket matching, syntax highlighting, etc.), which are inaccessible to visually-impaired programmers. While prior language-specific, downloadable tools have demonstrated benefits for the visually-impaired, we lack language-independent, cloud-based tools, both of which are critically needed. We present a new toolkit for building fully-accessible, browser-based programming environments for multiple languages. Given a parser that meets certain specifications, this toolkit will generate a block editor familiar to sighted users that also communicates the structure of a program using spoken descriptions, and allows for navigation using standard (accessible) keyboard shortcuts.

In modern software engineering, software plagiarism is widespread and uncurbed, developing plagiarism detection methods is imperative. Popular technologies of software plagiarism detection are mostly based on text, token and syntax tree. Among these plagiarism detection technologies, tree-based plagiarism detection technology can effectively detect the code which cannot be detected by the other two kinds of technologies. In this paper, we propose a more effective plagiarism detection algorithm based on abstract syntax tree (AST) by computing the hash values of the syntax tree nodes, and comparing them. In order to implement the algorithm more effectively, special measurement is taken to reduce the error rate when calculating the hash values of operations, especially the arithmetic operations like subtraction and division. Results of the test showed that the measurement is reliable and necessary. It performs well in the code comparison field, and is helpful in the area of protecting source code's copyright.

With the celebrated success of deep learning, some attempts to develop effective methods for detecting malicious PowerShell programs employ neural nets in a traditional natural language processing setup while others employ convolutional neural nets to detect obfuscated malicious commands at a character level. While these representations may express salient PowerShell properties, our hypothesis is that tools from static program analysis will be more effective. We propose a hybrid approach combining traditional program analysis (in the form of abstract syntax trees) and deep learning. This poster presents preliminary results of a fundamental step in our approach: learning embeddings for nodes of PowerShell ASTs. We classify malicious scripts by family type and explore embedded program vector representations.

Grasshopper is an Android application which teaches people JavaScript through a series of coding puzzles. As part of this application, we developed an AST-based JavaScript editor for the phone. The visual representation of the code is built using the AST as the source of truth, and user actions modify the AST rather than a code string, much like a block code editor. At the same time, we style this visual representation to look as similar as possible to syntax-highlighted, formatted JavaScript code in a text based editor. This paper outlines the system, the reasoning behind this design, and some early observations from users interacting with it.

暂无摘要（点击查看原文获取完整内容）

The increasing complexity of software applications and the necessity for minimizing software vulnerabilities has given rise to the use of machine learning techniques that can identify software vulnerabilities in source code. However, many of these techniques lack the accuracy needed for industrial practice. The contribution of this work is the novel use of an Abstract Syntax Tree Neural Network (ASTNN) to identify and classify software vulnerabilities in the Common Weakness Enumeration (CWE) types. We make two fundamental claims in this work. First, the use of an ASTNN performs better than prior machine learning neural network architectures. Second, the benchmark data set commonly used for machine learning vulnerability classification is flawed for this use. To illustrate these claims, we describe our ASTNN architecture and evaluate it with more than 44,000 test cases across 29 CWEs in the NIST Juliet Test Suite data set. Results show a minimum of 88% accuracy across all CWEs.

暂无摘要（点击查看原文获取完整内容）