CWA2: Proof of concept TOOLS
TOOLS: XBRL Validation
The objective of this document is to explain in short how to use the tools included in the Java Open Source packet XBRLValidation, to validate XBRL instance documents. Note these tools are part of a proof of concept, and still are in alpha state. These tools are therefore provided “AS IT IS”, without warranty of any kind.
Note all source codes (batch files, xml, command line, java, etc) end with two lines with three whitespaces an a dot (” .”). This is just only for compatibility with older navigators, and these two lines shouldn’t be considered part of the source code.
Prerequisites:
Before using these tools, you need:
- Java: If Java SE (Standard Edition) is not installed on your computer, download from Oracle and install it. This packet needs JDK version 1.7.0.25 (or newer). If Java SE is installed on your computer, verify version (by using “java –version”), and i needed download and install a newer one.
- Packet XBRLValidation_0_2_4.jar. This is the file, developed into the CEN WS XBRL project, where the classes have been packed. These classes are now under development, and are oriented just only as a proof of concept and didactic example, but not to be part of a production system. The source code will be publicly available under an open-source license (EUPL and other FLOSS licenses are being considered).
- XBRL Validator. This tools are pre-configured to be used with the Open Source XBRL platform “ARELLE“. You can download it from http://arelle.org/download/. This tool have been tested with Arelle version 2012-12-30 for Windows 64. In this document we suppose Arelle is installed in folder “c:\Program Files\Arelle”.
How XBRL Validator works
This tool launch a external XBRL instance documents validator, and capture the output of this validator. This output is filtered through a regular expression, the instance document is considered valid if after filtering there is no output from the validator. This tool can consider the standard output, the standard error, or both. The tool has been tested with Arelle (as previously mentioned). Currently we are in the process of testing it with other XBRL instance documents validators.
The tool uses a properties file to configure the command line and how to process the output of this command line. Java properties files have one or more key/s and a corresponding value/s. These are the keys of this properties file, and its sample values.
xbrlValidationCommandLineBeforeInstanceDocument=\"c:\\Program Files\\Arelle\\arelleCmdLine.exe\" --uiLang=EN –f xbrlValidationCommandLineAfterInstanceDocument= -v --logLevel=error regExpForFiltering=.* outputmanagement=StdoutAndStdErr . .
Where:
- xbrlValidationCommandLineBeforeInstanceDocument Is the part of the command line to be executed that is before the name of the instance document to be validated.
- xbrlValidationCommandLineAfterInstanceDocument Is the part of the command line to be executed that is after the name of the instance document to be validated.
- Outputmanagement Indicate which one of the output streams should be considered, the standard output (StdOut), the standard error (StdErr) or both (StdoutAndStdErr).
- regExpForFiltering A regular expression to filter the previously considered stream.
Launch XBRL Validator
To validate a file, use the command:
"c:\Program Files\Java\jdk1.7.0_25\bin\java.exe" -cp XBRLValidation.jar -Djdk.lang.Process.allowAmbigousCommands=true XBRLValidation.DoXBRLValidation PropertiesFile.properties InstanceDocument.xbrl . .
Where:
- PropertiesFile.properties: Is the properties file explained above.
- InstanceDocument.xbrl: Is the instance document to be validated.
- -Djdk.lang.Process.allowAmbigousCommands=true Is needed to fix a problem with the class Runtime. Somewhere in JDK 1.6, this class was chanted. This is the reason why JDK 1.7.0.25 is needed. There is another error, a spell error in allowAmbigousCommands, there is a missing u. This can be changed to allowAmbiguousCommands in future java versions. These changes are out of the scope of this tool, as we cannot have control over changes in standard java Runtime class.
Sample command line (line 1) and output on the screen of a wrong instance document:
"c:\Program Files\Java\jdk1.7.0_25\bin\java.exe" -cp XBRLValidation.jar -Djdk.lang.Process.allowAmbigousCommands=true XBRLValidation.DoXBRLValidation prop.properties ..\InstanceDocs\OK_rxt-20120930.xbrl True . .
Sample command line (line 1) and output on the screen of a correct instance document:
"c:\Program Files\Java\jdk1.7.0_25\bin\java.exe" -cp XBRLValidation.jar -Djdk.lang.Process.allowAmbigousCommands=true XBRLValidation.DoXBRLValidation prop.properties ..\InstanceDocs\BAD_2009031229.XBRL False [IOerror] ipp-gen-2008-01-01.xsd: file error: [Errno 2] No such file or directory: 'C:\\DEMOS\\InstanceDocs\\ipp-gen-2008-01-01.xsd' - BAD_2009031229.XBRL 3 [xbrl:schemaImportMissing] Instance facts missing schema definition: ipp-gen:InformeFinancieroGeneral - BAD_2009031229.XBRL 198 [xbrl:hrefFileNotFound] Href ipp-gen-2008-01-01.xsd file not found - BAD_2009031229.XBRL 3 . .
Source code:
Class DoXBRLValidation
package XBRLValidation;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.util.ArrayList;
import java.util.Properties;
import java.util.logging.Logger;
import java.util.regex.Matcher;
import java.util.regex.Pattern;
/**
* This class execute a command line to perform XBRL validation.
* There is a main method to allow the execution of this class from the command line.
* The main method gets as only input (in args) the name of the XBRL file to be validated.
* Also, it's is possible to call the method execValidator, to performs this validation from other paths.
*
*/
public class DoXBRLValidation {
public DoXBRLValidation() {
super();
}
private String xbrlValidationCommandLineBeforeInstanceDocument;
private String xbrlValidationCommandLineAfterInstanceDocument;
private String regExpForFiltering;
private String outputmanagement;
private RedirectOutput output;
private RedirectOutput error;
private static Logger LOGGER = Logger.getLogger("XBRLValidation.DoXBRLValidation");
/**
*Main method, to allow launch from command line.
* @param args There should be two args, the first with the properties file and the second one with the xbrl file to be validated.
* in boht cases, it's expected the full name of the files, including its paths.
*/
public static void main(String[] args) {
XBRLValidation.DoXBRLValidation validate = new XBRLValidation.DoXBRLValidation();
if (args.length == 2) {
validate.loadProperties(args[0]);
validate.execValidator(args[1]);
//validate.showResults();
validate.showFilteredResults();
} else {
System.out.println("Two arguments are expected, the first one with the properties file and the second one with the xbrl file to be validated.");
System.out.println("Sample content of a properties file (works for Windows 8 and Arelle:");
System.out.println("xbrlValidationCommandLineBeforeInstanceDocument=\\"c:\\Program Files\\Arelle\\arelleCmdLine.exe\" --uiLang=EN -f");
System.out.println("xbrlValidationCommandLineAfterInstanceDocument= -v --logLevel=error");
System.out.println("regExpForFiltering=.*");
System.out.println("outputmanagement=StdoutAndStdErr");
}
}
/**
* Load the property xbrlValidationCommandLine from the properties file specified as parameter:
* xbrlValidationCommandLineBeforeInstanceDocument= The command line of the xbrl validator BEFORE the instance document.
* In case of Arelle and windows use something as:
* c:\Program Files\Arelle\arelleCmdLine.exe" --uiLang=EN -v --loglevel=info -f
* xbrlValidationCommandLineAfterInstanceDocument= The command line of the xbrl validator AFTER the instance document.
* regExpForFiltering:
* outputmanagement
*
* @param propertiesFile Full name (with path) of the properties file.
*/
public void loadProperties(String propertiesFile) {
Properties prop = new Properties();
try {
InputStream in = new FileInputStream(propertiesFile); // getClass().getResourceAsStream(propertiesFile);
prop.load(in);
in.close();
xbrlValidationCommandLineBeforeInstanceDocument =
prop.getProperty("xbrlValidationCommandLineBeforeInstanceDocument");
xbrlValidationCommandLineAfterInstanceDocument =
prop.getProperty("xbrlValidationCommandLineAfterInstanceDocument");
//RegExpForFiltering: Regular expression for filtering
regExpForFiltering = prop.getProperty("regExpForFiltering");
//Should be: StdoutAndStdErr, StdOut or Stderr
outputmanagement = prop.getProperty("outputmanagement");
} catch (Exception ex) {
LOGGER.warning(ex.toString());
}
}
/**
* execValidator Execute the previously loaded XBRL validator in a new runtime environment.
*
* Execute the concatenation of strings xbrlValidationCommandLine and the parameter xbrlFileName and capture outputs.
*
* These outputs are stored in private methods output and error. Use getOutput() and getError() to access them.
*
* Both, output and error are instances of class RedirectOutput, and interested data (both outputs) are in private
* variable outputHistory (an ArrayList<String>). Use the getter method getOutputHistory() (of class RedirectOutput)
* in any of them to get the ArrayList (example, getOutput().getOutputHistory()).
*
* See method showResults for examples.
*
* @param xbrlFileName Xbrl file to be validated. Include full path in file name.
*/
private void execValidator(String xbrlFileName) {
try {
Runtime rt = Runtime.getRuntime();
String commandLine =
xbrlValidationCommandLineBeforeInstanceDocument + " \"" + xbrlFileName + "\" " + xbrlValidationCommandLineAfterInstanceDocument;
Process proc = rt.exec(commandLine);
setOutput(new RedirectOutput(proc.getInputStream())); //Add a FileOutputStream, to store output, fos.
getOutput().start();
setError(new RedirectOutput(proc.getErrorStream())); // Same to store error.
getError().start();
int exitVal = proc.waitFor();
} catch (Throwable t) {
t.printStackTrace();
}
}
/**
* Print on screen captured outputs form standard output and error output.
* Use as sample to access captured history.
*/
public void showResults() {
//Prints the output stream
if ((outputmanagement != null && outputmanagement.trim().length() > 0 &&
(outputmanagement.trim().toUpperCase().equals("STDOUTANDSTDERR") ||
outputmanagement.trim().toUpperCase().equals("STDOUT"))) &&
(getOutput() != null && getOutput().getOutputHistory() != null)) {
System.out.println("OUTPUT Stream: ");
for (String s : getOutput().getOutputHistory()) {
System.out.println(s);
}
}
//Prints the error stream, not very useful . . .
if ((outputmanagement != null && outputmanagement.trim().length() > 0 &&
(outputmanagement.trim().toUpperCase().equals("STDOUTANDSTDERR") ||
outputmanagement.trim().toUpperCase().equals("STDERR"))) &&
(getError() != null && getError().getOutputHistory() != null)) {
System.out.println("ERROR stream: ");
for (String s : getError().getOutputHistory()) {
System.out.println(s);
}
}
}
/**
* Print on screen the evaluation of the validation, and the validation according to:
* First, the output/error streams (or boths) are filtered by the specified regular expression.
* The regular expression is specified in the properties file, under key regExpForFiltering.
* In the properties file, key outputmanagement control if output/error stream (or both) are being considered.
* If the stream considered have some lines, the result of the validation is considered as "false"
* If the stream considered haven't lines, the result of the validation is considered as "true".
* This method prints in the screen the result of the validation, and then the content of the considered stream.
*/
public void showFilteredResults() {
ArrayList<String> res=filterRegExp(regExpForFiltering);
//System.out.println("Tras filtrar por: "+regExpForFiltering);
if (res.size()>0)
{
System.out.println("False");
for (String s:res) {
System.out.println(s);
}
}
else
System.out.println("True");
}
/**
* Filter using regular expression regexp.
*/
public ArrayList<String> filterRegExp(String regexp) {
ArrayList<String> result = new ArrayList<String>();
Pattern pattern = Pattern.compile(regexp);
if ((outputmanagement != null && outputmanagement.trim().length() > 0 &&
(outputmanagement.trim().toUpperCase().equals("STDOUTANDSTDERR") ||
outputmanagement.trim().toUpperCase().equals("STDOUT"))) &&
(getOutput() != null && getOutput().getOutputHistory() != null)) {
//System.out.println("OUTPUT Stream: ");
for (String s : getOutput().getOutputHistory()) {
if (regexp != null && regexp.trim().length() > 0) {
Matcher matcher = pattern.matcher(s);
if (matcher.matches())
result.add(s);
} else {
result.add(s);
}
}
}
//Prints the error stream, not very useful . . .
if ((outputmanagement != null && outputmanagement.trim().length() > 0 &&
(outputmanagement.trim().toUpperCase().equals("STDOUTANDSTDERR") ||
outputmanagement.trim().toUpperCase().equals("STDERR"))) &&
(getError() != null && getError().getOutputHistory() != null)) {
//System.out.println("ERROR stream: ");
for (String s : getError().getOutputHistory()) {
if (regexp != null && regexp.trim().length() > 0) {
Matcher matcher = pattern.matcher(s);
if (matcher.matches())
result.add(s);
} else {
result.add(s);
}
}
}
return result;
}
/*
* Getters && setters . . .
*/
public RedirectOutput getOutput() {
return output;
}
private void setOutput(RedirectOutput output) {
this.output = output;
}
public RedirectOutput getError() {
return error;
}
private void setError(RedirectOutput error) {
this.error = error;
}
}
.
.
Class RedirectOutput
package XBRLValidation;
import java.io.BufferedReader;
import java.io.InputStream;
import java.io.InputStreamReader;
import java.io.OutputStream;
import java.io.PrintWriter;
import java.util.ArrayList;
import java.util.logging.Logger;
/**
* This class captures standard output and standard error.
*
*/
public class RedirectOutput extends Thread {
private static Logger LOGGER = Logger.getLogger("XBRLValidation.RedirectOutput");
private InputStream is;
private OutputStream os;
/**
* outputHistory provides an ArrayList of string, being every one an output line from the command line.
* There is no setter for this variable, just only a getter to access from other objects.
*/
private ArrayList<String> outputHistory = new ArrayList<String>();
/**
* Constructor
*
* @param input Input stream, usually proc.getInputStream() or proc.getErrorStream()
*/
public RedirectOutput(InputStream input) {
this(input, null);
}
/**
* Constructor
*
* @param input Input stream, usually proc.getInputStream() or proc.getErrorStream()
* @param newOutput To redirect the output.
*/
public RedirectOutput(InputStream input, OutputStream newOutput) {
this.is = input;
this.os = newOutput;
}
/**
* Do the job. Read from the BufferedReader from the InputStream (this.is) every line, and store in outputHistory.
* Also, if an OutputStream is defined, redirect these lines (one by one) to this OutputStream.
*/
public void run() {
try {
PrintWriter pw = null;
if (os != null)
pw = new PrintWriter(os);
InputStreamReader isr = new InputStreamReader(is);
BufferedReader br = new BufferedReader(isr);
String linea = null;
linea = br.readLine();
while (linea != null) {
if (pw != null)
pw.println(linea);
outputHistory.add(linea);
linea = br.readLine();
}
if (pw != null)
pw.flush();
} catch (java.io.IOException ioe) {
LOGGER.warning(ioe.toString());
}
}
/**
* outputHistory getter. No public setter.
* @return
*/
public ArrayList<String> getOutputHistory() {
return outputHistory;
}
}
.
.
Class CreateFeedback
package XBRLValidation;
import java.io.FileNotFoundException;
import java.io.FileOutputStream;
import info.eurofiling.eu.fr.esrs.instancefeedback.*;
import java.io.File;
import java.io.FileInputStream;
import java.io.InputStream;
import java.security.DigestInputStream;
import java.security.MessageDigest;
import java.util.ArrayList;
import java.util.List;
import java.util.logging.Logger;
import javax.xml.bind.JAXBContext;
import javax.xml.bind.JAXBElement;
import javax.xml.bind.Marshaller;
public class CreateFeedback {
private static Logger LOGGER = Logger.getLogger("XBRLValidation.CreateFeedback");
public CreateFeedback() {
super();
}
public static void main(String[] args) {
CreateFeedback createFeedback = new CreateFeedback();
if (args.length == 3 || 1==1) {
String instanceDocument=args[0];
String propertiesFile=args[1];
String outputFile=args[2];
String[][] result=createFeedback.validationResultInBiDimensionalArray(instanceDocument, propertiesFile);
for (int i=0;i<result.length;i++) {
System.out.println("--->"+i+"--->");
System.out.println(" "+result[1][0]);
System.out.println(" "+result[1][1]);
System.out.println(" "+result[1][2]);
System.out.println(" "+result[1][3]);
System.out.println(" "+result[1][4]);
}
createFeedback.createXML_Feedback(instanceDocument, propertiesFile, outputFile);
} else {
System.out.println("Three arguments are expected, the first one with the properties file, the second one with the xbrl instance document to be validated and the third one with the xml output file.");
System.out.println("Sample content of a properties file (works for Windows 8 and Arelle:");
System.out.println("xbrlValidationCommandLineBeforeInstanceDocument=\"c:\\Program Files\\Arelle\\arelleCmdLine.exe\" --uiLang=EN -f");
System.out.println("xbrlValidationCommandLineAfterInstanceDocument= -v --logLevel=error");
System.out.println("regExpForFiltering=.*");
System.out.println("outputmanagement=StdoutAndStdErr");
}
}
/**
* This simple class computes SHA256 of a file.
*
* @param file Input file, for which one SHA256 should be computed
* @return String computation of SHA256 of the input file.
*/
public String getSHA256FromFile(String file) {
String res;
try{
FileInputStream fis=new FileInputStream(file);
MessageDigest md=MessageDigest.getInstance("SHA-256");
DigestInputStream dis=new DigestInputStream(fis,md);
while (dis.read()!=-1){;}//Could be more efficient, but is enought for demo purpose.
byte[] digest=dis.getMessageDigest().digest();
StringBuffer sb = new StringBuffer();
for(byte b : digest) {
sb.append(String.format("%02x", b));
}
res=sb.toString();
//System.out.println(res);
}
catch (Exception e){
res="";
LOGGER.warning(e.getMessage());
}
return res;
}
/**
* This method validates a XBRL instance document (using DoXBRLValidation class) and creates the output into an xml file.
*
* @param xbrlFile XBRL Instance document
* @param propertiesFile Properties file (see DoXBRLValidation class)
* @param outputFile Output XML file. The file will follow InstanceFeedback schema (http://www.eurofiling.info/eu/fr/esrs/InstanceFeedback/InstanceFeedback.xsd)
*/
public void createXML_Feedback(String xbrlFile, String propertiesFile, String outputFile) {
try {
DoXBRLValidation validate = new DoXBRLValidation();
validate.loadProperties(propertiesFile);
validate.execValidator(xbrlFile);
ArrayList<String> res = validate.getFilteredResults();
String shaValue=getSHA256FromFile(xbrlFile);
ClassLoader cl = info.eurofiling.eu.fr.esrs.instancefeedback.ObjectFactory.class.getClassLoader();
JAXBContext jc = JAXBContext.newInstance(InstanceFeedback.class);
InstanceFeedback feedback=new InstanceFeedback();
feedback.setInstanceName(xbrlFile);
feedback.setInstanceFilePath(xbrlFile); //Not clear which one will be at the end, Name or FilePath.
InstanceHashValueType hashValue=new InstanceHashValueType();
hashValue.setHashAlgorithm("http://www.w3.org/2001/04/xmlenc#sha256");
hashValue.setValue(shaValue);
feedback.setInstanceHashValue(hashValue);
if (res.size()>0)
{
feedback.setInstanceSuccessFlag(false);
List<ValidationPhase> validationPhaseList = feedback.getValidationPhase();
ValidationPhase val=new ValidationPhase();
ValidationErrorsType valErrors= new ValidationErrorsType();
List<ErrorType> validationErrorsType=valErrors.getValidationError();
for (String s:res) {
ErrorType valError=new ErrorType();
valError.setErrorCode("n/a");
valError.setErrorLocation("n/a");
ErrorSeverityType es=valError.getErrorSeverity();
valError.setErrorSeverity(es.ERROR);
valError.setErrorDescription(s);
validationErrorsType.add(valError);
}
val.setPhaseSuccessFlag(false);
val.setValidationPhaseType(ValidationPhaseType.XBRL_VALIDATION);
val.setValidationErrors(valErrors);
validationPhaseList.add(val);
}
else{
feedback.setInstanceSuccessFlag(true);
}
Marshaller m = jc.createMarshaller();
m.setProperty( Marshaller.JAXB_FORMATTED_OUTPUT, Boolean.TRUE );
m.marshal(feedback, new FileOutputStream(outputFile));
} catch (Exception e) {
LOGGER.warning(e.toString());
}
}
/**
* This method validates a XBRL instance document (using DoXBRLValidation class) and creates the output into bi-dimensional array of strings.
*
* @param xbrlFile XBRL Instance document
* @param propertiesFile Properties file (see DoXBRLValidation class)
*/
public String[][] validationResultInBiDimensionalArray(String xbrlFile, String propertiesFile) {
String[][] res;
DoXBRLValidation validate = new DoXBRLValidation();
validate.loadProperties(propertiesFile);
validate.execValidator(xbrlFile);
ArrayList<String> resValidation = validate.getFilteredResults();
if (resValidation.size()>0)
{
res=new String[resValidation.size()][5];
int i=0;
for (String s:resValidation) {
res[i][0]="XBRLValidation";
res[i][1]="n/a";
res[i][2]="n/a";
res[i][3]=s;
res[i][4]="Error";
i++;
}
}
else{
res=null;
}
return res;
}
}
.
.
Question? Please feel free to contact us: cenwsxbrl at gmail.com