Smart Autocomplete Add-on (Experimental)

Suppose you need to fine tune a model to learn to apply markup to your text.

{"prompt": "<prompt text>", "completion": "<sample of generated text>"}

For more details about this format, see: https://beta.openai.com/docs/guides/fine-tuning/prepare-training-data.

First, some constants are defined and then the root matching template that scans a folder that contains DITA topics is created:

  <!-- Method "text" because we generate JSON -->
  <xsl:output method="text" omit-xml-declaration="yes"/>
  
  <!-- Paragraphs that are smaller than this limit are processed. -->
  <xsl:variable name="limit" select="500"/>
  <!-- Open AI recommends adding terminators to the prompt and completion -->
  <xsl:variable name="prompt-terminator" select="' ### '"/>
  <xsl:variable name="completion-terminator" select="'$$$'"/>  

  <xsl:template match="/">
      <xsl:for-each select="collection('file:/C:/path-to-topics-folder')">
        <xsl:call-template name="process-file"/>
      </xsl:for-each>
  </xsl:template>

Next, the template that processes each of the topic files is defined. Only the <p> and <li> elements are selected in this example, and for each of them, a prompt-completion pair is generated.

<xsl:template name="process-file">
    <xsl:message>Processing <xsl:value-of select="document-uri(.)"/></xsl:message>
    <xsl:for-each select=".//*[contains(@class, ' topic/p ') or 
                               contains(@class, ' topic/li ')]">      
      <xsl:variable name="text" select="oxy:clean(.)"/>
      
      <xsl:variable name="text-with-markup">
        <xsl:apply-templates mode="serialize-markup"/>
      </xsl:variable>           
      <xsl:variable name="text-with-markup" select="oxy:clean($text-with-markup)"/>
      
      <xsl:if test="string-length($text)&lt;$limit and 
                    string-length($text-with-markup)&lt;$limit and
                    string-length($text)>0 and
                    string-length($text-with-markup)>0 and 
                    string-length($text) != string-length($text-with-markup) 
                    " xml:space="preserve">
{"prompt":"<xsl:value-of select="$text" 
disable-output-escaping="yes"/><xsl:value-of 
select="$prompt-terminator"/>", "completion":" <xsl:value-of 
select="$text-with-markup" 
disable-output-escaping="yes"/><xsl:value-of 
select="$completion-terminator"/>"}</xsl:if>
    </xsl:for-each>
  </xsl:template>

Since the format is JSON, the prompt and completion strings need to be cleaned up by escaping the quotes and apostrophes. This function is used from the above template.

  <xsl:function name="oxy:clean">
    <xsl:param name="txt"/>
    <xsl:variable name="apos">'</xsl:variable>
    <xsl:variable name="quot">"</xsl:variable>
    
    <xsl:variable name="txt">        
      <xsl:value-of select="replace(normalize-space($txt), $quot, $apos)"/>
    </xsl:variable>
    <xsl:variable name="txt">        
      <xsl:value-of select="replace($txt,'\\', '\\\\')"/>
    </xsl:variable>
    <xsl:sequence select="$txt"/>
  </xsl:function>

Next, to serialize the markup from a paragraph, all the inlines of interest are matched and the element names and attributes are emitted for each of them:

  <!--
    Match the inline elements and serializes them. 
   -->
  <xsl:template match="*[
      contains(@class, ' ui-d/uicontrol ') or 
      contains(@class, ' ui-d/menucascade ') or
      contains(@class, ' ui-d/shortcut ') or
      contains(@class, ' topic/i ') or      
      contains(@class, ' topic/b ') or      
      contains(@class, ' sw-d/filepath ') or
      contains(@class, ' pr-d/codeph ') or
      contains(@class, ' pr-d/codeblock ')
      ]" mode="serialize-markup" >
    <!-- Only non empty elements. -->
    <xsl:text>&lt;</xsl:text>
    <xsl:value-of select="local-name()"/>
    <xsl:for-each select="@*">
      <xsl:if test="local-name() != 'class' and local-name() != 'id'">
        <xsl:text> </xsl:text>
        <xsl:value-of select="local-name()"/>
        <xsl:text>='</xsl:text>
        <xsl:value-of select="."/>
        <xsl:text>'</xsl:text>    
      </xsl:if>
    </xsl:for-each>
    <xsl:text>&gt;</xsl:text>
    <xsl:apply-templates mode="serialize-markup"/>
    <xsl:text>&lt;/</xsl:text>    
    <xsl:value-of select="local-name()"/>
    <xsl:text>&gt;</xsl:text>    
  </xsl:template>

To apply this stylesheet, create an XSLT transformation scenario in Oxygen XML Editor. You can choose any well-formed XML file as the input (the stylesheet scans a folder using the collection function, so this input will be ignored) and specify the output file.