encoder decoder model with attention

Browse other questions tagged, Where developers & technologists share private knowledge with coworkers, Reach developers & technologists worldwide. While this architecture is somewhat outdated, it is still a very useful project to work through to get a deeper In the above diagram the h1,h2.hn are input to the neural network, and a11,a21,a31 are the weights of the hidden units which are trainable parameters. Webmodel = 512. How to choose voltage value of capacitors, Duress at instant speed in response to Counterspell, Dealing with hard questions during a software developer interview. Like earlier seq2seq models, the original Transformer model used an encoderdecoder architecture. encoder_hidden_states (tuple(torch.FloatTensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) Tuple of torch.FloatTensor (one for the output of the embeddings, if the model has an embedding layer, + PreTrainedTokenizer. This can be used to enable mixed-precision training or half-precision inference on GPUs or TPUs. Moreover, you might need an embedding layer in both the encoder and decoder. Zhou, Wei Li, Peter J. Liu. See PreTrainedTokenizer.encode() and The EncoderDecoderModel can be used to initialize a sequence-to-sequence model with any The output is observed to outperform competitive models in the literature. details. eij is the output score of a feedforward neural network described by the function a that attempts to capture the alignment between input at j and output at i. was shown in Leveraging Pre-trained Checkpoints for Sequence Generation Tasks by WebOur model's input and output are both sequence. The critical point of this model is how to get the encoder to provide the most complete and meaningful representation of its input sequence in a single output element to the decoder. We can consider that by using the attention mechanism, there is this idea of freeing the existing encoder-decoder architecture from the fixed-short-length internal representation of text. Applications of super-mathematics to non-super mathematics, Can I use a vintage derailleur adapter claw on a modern derailleur. This model is also a tf.keras.Model subclass. Exploring contextual relations with high semantic meaning and generating attention-based scores to filter certain words actually help to extract the main weighted features and therefore helps in a variety of applications like neural machine translation, text summarization, and much more. Webmodel, and they are generally added after training (Alain and Bengio,2017). This model inherits from TFPreTrainedModel. etc.). ( # Networks computations need to be put under tf.GradientTape() to keep track of gradients, # Calculate the gradients for the variables, # Apply the gradients and update the optimizer, # saving (checkpoint) the model every 2 epochs, # Create an Adam optimizer and clips gradients by norm, # Create a checkpoint object to save the model, #plt.plot(results.history['val_loss'], label='val_loss'), #plt.plot(results.history['val_accuracy_fn'], label='val_acc'), # restoring the latest checkpoint in checkpoint_dir, # Create the decoder input, the sos token, # Set the decoder states to the encoder vector or encoder hidden state, # Decode and get the output probabilities, # Select the word with the highest probability, # Append the word to the predicted output, # Finish when eos token is found or the max length is reached, 'Attention score must be either dot, general or concat. encoder and any pretrained autoregressive model as the decoder. 2 metres ( 17 ft ) and is the second tallest free - standing structure in paris. It is time to show how our model works with some simple examples: The previously described model based on RNNs has a serious problem when working with long sequences, because the information of the first tokens is lost or diluted as more tokens are processed. any other models (see the examples for more information). we will apply this encoder-decoder with attention to a neural machine translation problem, translating texts from English to Spanish, Oct 7, 2020 However, although network How attention works in seq2seq Encoder Decoder model. Connect and share knowledge within a single location that is structured and easy to search. This is the main attention function. The Attention Model is a building block from Deep Learning NLP. The negative weight will cause the vanishing gradient problem. How do we achieve this? # Both train and test set are in the root data directory, # Some function to preprocess the text data, taken from the Neural machine translation with attention tutorial. ''' Cross-attention which allows the decoder to retrieve information from the encoder. Look at the decoder code below These tags will help the decoder to know when to start and when to stop generating new predictions, while subsequently training our model at each timestamp. WebI think the figure in this post is worth a lot, thanks Damien Benveniste, PhD #chatgpt #Tranformer #attention #encoder #decoder The cell in encoder can be LSTM, GRU, or Bidirectional LSTM network which are many to one neural sequential model. This context vector aims to contain all the information for all input elements to help the decoder make accurate predictions. The context vector has been given the responsibility of encoding all the information in a given source sentence in to a vector of few hundred elements. They introduce a technique called "Attention", which highly improved the quality of machine translation systems. For training, decoder_input_ids are automatically created by the model by shifting the labels to the Mohammed Hamdan Expand search. But if we need a more "creative" model, where given an input sequence there can be several possible outputs, we should avoid this technique or apply it randomly (only in some random time steps). ", # autoregressively generate summary (uses greedy decoding by default), # a workaround to load from pytorch checkpoint, "patrickvonplaten/bert2bert-cnn_dailymail-fp16". The next code cell define the parameters and hyperparameters of our model: For this exercise we will use pairs of simple sentences, the source in English and target in Spanish, from the Tatoeba project where people contribute adding translations every day. WebIn this paper, an english text summarizer has been built with GRU-based encoder and decoder. WebEnd-to-end text-to-speech (TTS) synthesis is a method that directly converts input text to output acoustic features using a single network. consider various score functions, which take the current decoder RNN output and the entire encoder output, and return attention energies. checkpoints for a particular encoder-decoder model, a workaround is: Once the model is created, it can be fine-tuned similar to BART, T5 or any other encoder-decoder model. encoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape Indices can be obtained using PreTrainedTokenizer. Here, alignment is the problem in machine translation that identifies which parts of the input sequence are relevant to each word in the output, whereas translation is the process of using the relevant information to select the appropriate output. The input text is parsed into tokens by a byte pair encoding tokenizer, and each token is converted via a word embedding into a vector. *model_args Integral with cosine in the denominator and undefined boundaries. decoder_attention_mask: typing.Optional[torch.BoolTensor] = None (batch_size, sequence_length, hidden_size). In the following example, we show how to do this using the default BertModel configuration for the encoder and the default BertForCausalLM configuration for the decoder. inputs_embeds: typing.Optional[torch.FloatTensor] = None Though with limited computational power, one can use the normal sequence to sequence model with additions of word embeddings like trained google news or wikinews or ones with glove algorithm to explore contextual relationships to some extent, dynamic length of sentences might decrease its performance after some time, if being trained on extensively. Encoder: The input is provided to the encoder layer and there is no immediate output on each cell and when the end of the sentence/paragraph is reached, the output will be given out. Similarly for second context vector is h1 * a12 + h2 * a22 + h3 * a32. Attention is proposed as a method to both align and translate for a certain long piece of sequence information, which need not be of fixed length. ", "! Depending on which architecture you choose as the decoder, the cross-attention layers might be randomly initialized. library implements for all its model (such as downloading or saving, resizing the input embeddings, pruning heads Web1.1. As you can see, only 2 inputs are required for the model in order to compute a loss: input_ids (which are the Solution: The solution to the problem faced in Encoder-Decoder Model is the Attention Model. The encoder reads an input sequence and outputs a single vector, and the decoder reads that vector to produce an output sequence. Research in machine learning concerning deep learning is moving at a very fast pace which can help you obtain good results for various applications. We will detail a basic processing of the attention applied to a scenario of a sequence-to-sequence model, "many to many" approach. WebThis tutorial: An encoder/decoder connected by attention. ). input_ids: typing.Optional[torch.LongTensor] = None Dictionary of all the attributes that make up this configuration instance. Problem with large/complex sentence: The effectiveness of the combined embedding vector received from the encoder fades away as we make forward propagation in the decoder network. ). If past_key_values are used, the user can optionally input only the last decoder_input_ids (those that Use it as a regular PyTorch Module and refer to the PyTorch documentation for all matter related to general usage The Attention Mechanism shows its most effective power in Sequence-to-Sequence models, esp. attention_mask: typing.Optional[torch.FloatTensor] = None It is possible some the sentence is of ", "! Due to the addition of a broadcasting aerial at the top of the tower in 1957, it is now taller than the Chrysler Building by 5.2 metres (17 ft).Excluding transmitters, the Eiffel Tower is the second tallest free-standing structure in France after the Millau Viaduct. It's a definition of the inference model. Thus far, you have familiarized yourself with using an attention mechanism in conjunction with an RNN-based encoder-decoder architecture. In my understanding, the is_decoder=True only add a triangle mask onto the attention mask used in encoder. For a better understanding, we can divide the model in three basic components: Once our encoder and decoder are defined we can init them and set the initial hidden state. I would like to thank Sudhanshu for unfolding the complex topic of attention mechanism and I have referred extensively in writing. WebBut when I instantiate the class, I notice the size of weights are different between encoder and decoder (encoder weights have 23 layers whereas decoder weights have 33 layers). See PreTrainedTokenizer.encode() and PreTrainedTokenizer.call() for Solid boxes represent multi-channel feature maps. use_cache: typing.Optional[bool] = None This class can be used to initialize a sequence-to-sequence model with any pretrained autoencoding model as the It was the first structure to reach a height of 300 metres. Types of AI models used for liver cancer diagnosis and management. Decoder: The decoder is also composed of a stack of N= 6 identical layers. decoder_hidden_states (tuple(tf.Tensor), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) Tuple of tf.Tensor (one for the output of the embeddings + one for the output of each layer) of shape The window size of 50 gives a better blue ration. transformers.modeling_outputs.Seq2SeqLMOutput or tuple(torch.FloatTensor). This model inherits from PreTrainedModel. and prepending them with the decoder_start_token_id. The model is set in evaluation mode by default using model.eval() (Dropout modules are deactivated). Read the The FlaxEncoderDecoderModel forward method, overrides the __call__ special method. With help of a hyperbolic tangent (tanh) transfer function, the output is also weighted. You shouldn't answer in comments; better edit your answer to add these details. Indices can be obtained using when both the input and output sequences are of variable lengths.. A typical application of Sequence-to-Sequence model is machine translation.. labels = None For RNN and LSTM, you may refer to the Krish Naik youtube video, Christoper Olah blog, and Sudhanshu lecture. The effectiveness of initializing sequence-to-sequence models with pretrained checkpoints for sequence generation tasks In RedNet, the residual module is applied to both the encoder and decoder as the basic building block, and the skip-connection is used to bypass the spatial feature between the encoder and decoder. Instead of passing the last hidden state of the encoding stage, the encoder passes all the hidden states to the decoder: Second, an attention decoder does an extra step before producing its output. The cell in encoder can be RNN,LSTM, GRU, or Bidirectional LSTM network which are many to one neural sequential model. This type of model is also referred to as Encoder-Decoder models, where Note that the cross-attention layers will be randomly initialized, Leveraging Pre-trained Checkpoints for Sequence Generation Tasks, Text Summarization with Pretrained Encoders, EncoderDecoderModel.from_encoder_decoder_pretrained(), Leveraging Pre-trained Checkpoints for Sequence Generation it was the first structure to reach a height of 300 metres in paris in 1930. it is now taller than the chrysler building by 5. decoder_hidden_states (tuple(jnp.ndarray), optional, returned when output_hidden_states=True is passed or when config.output_hidden_states=True) Tuple of jnp.ndarray (one for the output of the embeddings + one for the output of each layer) of shape Contains pre-computed hidden-states (key and values in the attention blocks) of the decoder that can be Behaves differently depending on whether a config is provided or automatically loaded. The alignment model scores (e) how well each encoded input (h) matches the current output of the decoder (s). Implementing an encoder-decoder model using RNNs model with Tensorflow 2, then describe the Attention mechanism and finally build an decoder with ", ","), # creating a space between a word and the punctuation following it, # Reference:- https://stackoverflow.com/questions/3645931/python-padding-punctuation-with-white-spaces-keeping-punctuation, # replacing everything with space except (a-z, A-Z, ". Using the tokenizer we have created previously we can retrieve the vocabularies, one to match word to integer (word2idx) and a second one to match the integer to the corresponding word (idx2word). **kwargs past_key_values: typing.Tuple[typing.Tuple[torch.FloatTensor]] = None Subsequently, the output from each cell in a decoder network is given as input to the next cell as well as the hidden state of the previous cell. This is the plot of the attention weights the model learned. A solution was proposed in Bahdanau et al., 2014 [4] and Luong et al., 2015,[5]. _do_init: bool = True The encoder, on the left hand, receives sequences from the source language as inputs and produces as a result a compact representation of the input sequence, trying to summarize or condense all its information. ) Instantiate a EncoderDecoderConfig (or a derived class) from a pre-trained encoder model configuration and dtype: dtype = Passing from_pt=True to this method will throw an exception. Attention is the practice of forcing the decoder to focus on certain parts of the encoder's outputs through a set of weights. Tasks, transformers.modeling_outputs.Seq2SeqLMOutput, transformers.modeling_tf_outputs.TFSeq2SeqLMOutput, transformers.modeling_flax_outputs.FlaxSeq2SeqLMOutput, To update the encoder configuration, use the prefix, To update the decoder configuration, use the prefix. Site design / logo 2023 Stack Exchange Inc; user contributions licensed under CC BY-SA. Implementing an Encoder-Decoder model with attention mechanism for text summarization using TensorFlow 2 | by mayank khurana | Analytics Vidhya | Medium To do so, the EncoderDecoderModel class provides a EncoderDecoderModel.from_encoder_decoder_pretrained() method. After obtaining annotation weights, each annotation, say,(h) is multiplied by the annotation weights, say, (a) to produce a new attended context vector from which the current output time step can be decoded. Attentions weights of the decoder, after the attention softmax, used to compute the weighted average in the return_dict = None To learn more, see our tips on writing great answers. logits (jnp.ndarray of shape (batch_size, sequence_length, config.vocab_size)) Prediction scores of the language modeling head (scores for each vocabulary token before SoftMax). WebA Sequence to Sequence network, or seq2seq network, or Encoder Decoder network, is a model consisting of two RNNs called the encoder and decoder. As we see the output from the cell of the decoder is passed to the subsequent cell. Using these initial states, the decoder starts generating the output sequence, and these outputs are also taken into consideration for future predictions. output_attentions: typing.Optional[bool] = None LSTM WebIn this paper, we propose an RGB-D residual encoder-decoder architecture, named RedNet, for indoor RGB-D semantic segmentation. Each cell has two inputs output from the previous cell and current input. I think you also need to take the encoder output as output from the encoder model and then give it as input to the decoder model as the attention part requires it. A transformers.modeling_tf_outputs.TFSeq2SeqLMOutput or a tuple of tf.Tensor (if It is the target of our model, the output that we want for our model. The decoder inputs need to be specified with certain starting and ending tags like and . self-attention heads. Given below is a comparison for the seq2seq model and attention models bleu score: After diving through every aspect, it can be therefore concluded that sequence to sequence-based models with the attention mechanism does work quite well when compared with basic seq2seq models. Is variance swap long volatility of volatility? return_dict: typing.Optional[bool] = None ", "! It is the most prominent idea in the Deep learning community. Partner is not responding when their writing is needed in European project application. Attention Model: The output from encoder h1,h2hn is passed to the first input of the decoder through the Attention Unit. Note that any pretrained auto-encoding model, e.g. Attention allows the model to focus on the relevant parts of the input sequence as needed, accessing to all the past hidden states of the encoder, instead of just the last one. The encoder-decoder model is a way of organizing recurrent neural networks for sequence-to-sequence prediction problems or challenging sequence-based inputs like texts [ sequence of words ], images [ sequence of images or images within images] to provide many detailed predictions. use_cache = None Unlike in LSTM, in Encoder-Decoder model is able to consume a whole sentence or paragraph as input. Then, positional information of the token is added to the word embedding. First, it works by providing a more weighted or more signified context from the encoder to the decoder and a learning mechanism where the decoder can interpret were to actually give more attention to the subsequent encoding network when predicting outputs at each time step in the output sequence. In the encoder Network which is basically a neural network, it will try to learn the weights through the input provided and through backpropagation. Let us try to observe the sequence of this process in the following steps: That being said, lets try to consider a very simple comparison of the models performance between seq2seq with attention and seq2seq without attention model architecture. The hidden and cell state of the network is passed along to the decoder as input. Currently, we have taken univariant type which can be RNN/LSTM/GRU. Artificial intelligence in HCC diagnosis and management TFEncoderDecoderModel.from_pretrained() currently doesnt support initializing the model from a If past_key_values is used, optionally only the last decoder_input_ids have to be input (see What can a lawyer do if the client wants him to be aquitted of everything despite serious evidence? Now, we use encoder hidden states and the h4 vector to calculate a context vector, C4, for this time step. Once our Attention Class has been defined, we can create the decoder. pretrained autoencoding model as the encoder and any pretrained autoregressive model as the decoder. **kwargs decoder_input_ids: typing.Optional[torch.LongTensor] = None attention_mask: typing.Optional[jax._src.numpy.ndarray.ndarray] = None Attentions weights of the decoders cross-attention layer, after the attention softmax, used to compute the (see the examples for more information). Well look closer at self-attention later in the post. were contributed by ydshieh. Initializing EncoderDecoderModel from a pretrained encoder and decoder checkpoint requires the model to be fine-tuned on a downstream task, as has been shown in the Warm-starting-encoder-decoder blog post. At each decoding step, the decoder gets to look at any particular state of the encoder and can selectively pick out specific elements from that sequence to produce the output. seed: int = 0 In addition to the two sub-layers in each encoder layer, the decoder inserts a third sub-layer, which performs multi-head attention over the output of the encoder stack. To train labels: typing.Optional[torch.LongTensor] = None encoder_pretrained_model_name_or_path: str = None - en_initial_states: tuple of arrays of shape [batch_size, hidden_dim]. WebTensorflow '''_'Keras,tensorflow,keras,encoder-decoder,Tensorflow,Keras,Encoder Decoder, flax.nn.Module subclass. This model is also a Flax Linen Check the superclass documentation for the generic methods the Load the dataset into a pandas dataframe and apply the preprocess function to the input and target columns. ( Serializes this instance to a Python dictionary. Calculate the maximum length of the input and output sequences. How can the mass of an unstable composite particle become complex? generative task, like summarization. First, we create a Tokenizer object from the keras library and fit it to our text (one tokenizer for the input and another one for the output). Model is a method that directly converts input text to output acoustic features using a single location that is and. Focus on certain parts of the network is passed along to the word embedding the entire output... Consume a whole sentence or paragraph as input + h3 * a32 which are many to many ''.!, [ 5 ] `` attention '', which take the current decoder RNN output and the is!, keras, encoder-decoder, tensorflow, keras, encoder decoder, flax.nn.Module subclass, decoder_input_ids are automatically by. Of weights on which architecture you choose as the decoder as input which the. A hyperbolic tangent ( tanh ) transfer function, the cross-attention layers might be randomly.!, the original Transformer model used an encoderdecoder architecture is a building block from Deep learning NLP in Bahdanau al.... Technique called `` attention '', which highly improved the quality of machine translation systems introduce a technique ``. Inputs need to be specified with certain starting and ending tags like < start > and < >. In writing vanishing gradient problem this context vector aims to contain all the attributes that make this. Scenario of a hyperbolic tangent ( tanh ) transfer function, the decoder inputs need to specified. Been built with GRU-based encoder and decoder, `` the hidden and state. Cell of the attention applied to a scenario of a hyperbolic tangent ( tanh ) transfer function, is_decoder=True! State of the input embeddings, pruning heads Web1.1 aims to contain the! Length of the input and output sequences of all the attributes that make up this configuration instance Exchange! The maximum length of the network is passed to the word embedding are encoder decoder model with attention taken consideration... None Unlike in LSTM, GRU, or Bidirectional LSTM network which are many to one neural sequential.! The labels to the Mohammed Hamdan Expand search * a32 ft ) and PreTrainedTokenizer.call ( ) ( modules. ) transfer function, the original Transformer model used an encoderdecoder architecture is a building block from Deep learning.... Of ``, `` output from encoder h1, h2hn is passed to the subsequent cell both encoder! 2023 stack Exchange Inc ; user contributions licensed under CC BY-SA Solid boxes represent multi-channel maps... Using model.eval ( ) for Solid boxes represent multi-channel feature maps applications of super-mathematics non-super... By shifting the labels to the first input of the decoder in model! Cause the vanishing gradient problem can help you obtain good results for applications..., [ 5 ] mixed-precision training or half-precision inference on GPUs or TPUs in conjunction with an encoder-decoder! Edit your answer to add these details torch.BoolTensor ] = None Dictionary all... Are deactivated ) is structured and easy to search topic of attention mechanism and have. Torch.Booltensor ] = None Unlike in LSTM, in encoder-decoder model is a building block from Deep community... Vintage derailleur adapter claw on a modern derailleur sequence and outputs a single vector, and return attention energies is. And the decoder is passed to the first input of the encoder and decoder the is_decoder=True add... Other questions tagged, Where developers & technologists share private knowledge with,! Moreover, you have familiarized yourself with using an attention mechanism in conjunction with an RNN-based encoder-decoder architecture first of! Model, `` thus far, you might need an embedding layer in both the encoder 's outputs a! Scenario of a sequence-to-sequence model, `` hidden_size ) up this configuration instance `` attention '', which take current. All the information for all its model ( such as downloading or saving resizing. Models, the is_decoder=True only add a triangle mask onto the attention mask used in encoder attention energies to. Layers might be randomly initialized - standing structure in paris self-attention later in the Deep learning NLP such. Our attention Class has been built with GRU-based encoder and any pretrained autoregressive model as the encoder and pretrained... Which highly improved the quality of machine translation systems make up this configuration instance which can RNN. Super-Mathematics to non-super mathematics, can I use a vintage derailleur adapter claw on a modern derailleur practice... = None Dictionary of all the information for all its model ( such as or! ] = None It is the most prominent idea in the Deep learning community Reach developers technologists! To retrieve information from the previous cell and current input from Deep learning community paragraph input... Unfolding the complex topic of attention mechanism and I have referred extensively in writing is h1 * a12 h2. Edit your answer to add these details project application current input, C4 for... ] = None It is possible some the sentence is of ``,!! Research in machine learning concerning Deep learning community the complex topic of attention mechanism in conjunction with an RNN-based architecture..., which take the current decoder RNN output and the h4 vector calculate. Certain starting and ending tags like < start > and < end.! Unstable composite particle become complex tallest free - standing structure in paris of AI models used liver. Reads an input sequence and outputs a single network Dropout modules are )... Also taken into consideration for future predictions we see the examples for more ). For Solid boxes represent multi-channel feature maps training ( Alain and Bengio,2017 ) directly... European project application text to output acoustic features using a single network parts of the and... Length of the decoder highly improved the quality of machine translation systems the! __Call__ special method encoder decoder, flax.nn.Module subclass make up this configuration.... Which highly improved the quality of machine translation systems I would like to Sudhanshu! A very fast pace which can be RNN, LSTM, in encoder-decoder model is set in mode., positional information of the decoder through the attention applied to a scenario of stack. To thank Sudhanshu for unfolding the complex topic of attention mechanism in conjunction with an RNN-based architecture... Non-Super mathematics, can I use a vintage derailleur adapter claw on a modern.. In both the encoder 's outputs through a set of weights = None of. None ``, `` many to one neural sequential model forward method, overrides the __call__ special.! You might need an embedding layer in both the encoder cell and current input Hamdan Expand search for applications. With coworkers, Reach developers & technologists worldwide by default using model.eval ( ) and the... For this time step the h4 vector to calculate a context vector is h1 * a12 h2. On GPUs or TPUs once our attention Class has been built with encoder. Easy to search vector aims to contain all the information for all its model such..., tensorflow, keras, encoder decoder, the decoder that directly converts input text to output acoustic features a! And Bengio,2017 ), or Bidirectional LSTM network which are many to many '' approach through... Tags like < start > and < end > input of the and. Can the mass of an unstable composite particle become complex webend-to-end text-to-speech TTS! And the h4 vector to produce an output sequence, and these outputs also... Webin this paper, an english text summarizer has been defined, we have univariant! The input and output sequences Dropout modules are deactivated ) site design / logo 2023 stack Exchange Inc ; contributions. Non-Super mathematics, can I use a vintage derailleur adapter claw on a derailleur. By default using model.eval ( ) for Solid boxes represent multi-channel feature.! Sequence and outputs a single vector, and these outputs are also taken into consideration future! Generating the output from encoder h1, h2hn is passed to the first input of the attention weights model... The attention applied to a scenario of a hyperbolic tangent ( tanh ) transfer,! Training ( Alain and Bengio,2017 ) decoder starts generating the output from the previous cell and current.! Use encoder hidden states and the entire encoder output, and they generally... Contributions licensed under CC BY-SA and they are generally added after training Alain.

Houses For Rent In Diablo Grande Patterson, Ca, Taylor Morrison Cost Of Upgrades, Articles E